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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the  Office  of 
the  Director,  Industrial  Capabilities  and  Assessments,  Under  Secretary  of  Defense  for  Acqui¬ 
sition  and  Technology,  under  the  task  entitled  Integrated  Diagnostics  and  Improved  Affordabil¬ 
ity  for  Weapon  Support  Systems.  It  fulfills  the  following  task  objective:  ‘In  conjunction  with 
an  industry  review  forum,  identify  opportunities  and  develop  concepts  and  demonstration 
implementation  approaches  for  improving  integrated  diagnostics.”  This  paper  documents  the 
activities  and  results  of  the  Joint  Service  Integrated  Diagnostics  Workshop  held  at  IDA  on 
August  8, 1996,  and  provides  an  IDA  study  team’s  analyses  of  the  results. 

The  following  IDA  research  staff  members  were  reviewers  of  this  document:  Dr.  Alfred 
E.  Brenner,  Dr.  Dennis  W.  Fife,  Dr.  Richard  J.  Ivanetich,  Mr.  Terry  Mayfield,  Mr.  Michael  S. 
Nash,  and  Dr.  Danny  L.  Reed. 
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EXECUTIVE  SUMMARY 


On  August  8,  1996,  the  Institute  for  Defense  Analyses  (IDA)  hosted  a  Joint  Service 
Integrated  Diagnostics  Workshop  under  the  auspices  of  the  Office  of  the  Director,  Industrial 
Capabilities  and  Assessments,  Under  Secretary  of  Defense  for  Acquisition  and  Technology. 
The  participants  were  from  U.S.  government  organizations  and  included  representatives  from 
the  technology  development,  acquisition,  and  support  functional  areas  of  the  Services  (Army, 
Air  Force,  Navy,  and  Marines).  This  paper  documents  the  activities  and  results  of  the  workshop 
and  provides  an  IDA  smdy  team’s  independent  analyses  of  the  workshop  results. 

Background 

A  necessary  step  in  the  maintenance  and  repair  process  of  weapon  systems  is  investi¬ 
gating  the  nature  or  cause  of  hardware  and  software  anomalies  inhibiting  normal  operation. 
Integrated  diagnostics  represents  a  systems  approach  where  integrating  diagnostic  elements 
creates  a  total  diagnostic  capability  that  outperforms  individual  support  and  maintenance  tools 
operating  alone.  While  specific  benefits  of  robust  integrated  diagnostics  capabilities  will  vaiy 
by  application,  reported  benefits  include  greater  operational  readiness,  improved  systems  con¬ 
fidence,  improved  availability,  reduced  maintenance  work  loads,  and  reduced  life-cycle  costs. 

While  some  progress  has  been  made  in  integrating  diagnostic  elements,  extensive  inte¬ 
gration  into  systems,  once  fielded  or  as  they  become  legacy  systems,  has  often  been  difficult  to 
justify.  To  begin  with,  the  development  of  diagnostic  elements  takes  a  long  time  for  new  weap¬ 
on  systems  and  often  occurs  near  the  end  of  the  weapon  development  period.  Rarely  are  inte¬ 
grated  diagnostic  improvements  to  legacy  systems  justified  as  stand-alone  initiatives.  Instead, 
legacy  system  diagnostic  improvements — ^when  they  do  occur — ^tend  to  be  secondary  benefits 
of  system  modifications  originally  made  to  improve  performance.  Also  the  degree  and  types  of 
diagnostic  element  integration  are  severely  limited  by  infi-astructures  already  put  in  place  for 
legacy  systems. 

Approach 

Workshop.  In  preparation  for  the  workshop,  IDA  asked  would-be  attendees  to  submit 
descriptions  of  test  and  diagnostics  problems  and  their  proposed  solutions.  At  the  workshop. 


participants  were  divided  into  two  working  groups:  Legacy  Systems  and  Cross-Cutting.  The 
Legacy  Systems  Working  Group  was  asked  to  identify  test  and  diagnostics  problems  and  poten¬ 
tial  solutions  for  legacy  systems;  the  Cross-Cutting  Working  Group  was  asked  to  identify  test 
and  diagnostics  problems  that  crossed  domains  as  well  as  potential  solutions  to  these  problems. 

IDA  analysis  of  workshop  results.  The  IDA  study  team  created  a  set  of  five  questions 
and  used  them  to  organize  and  assess  issues,  problems,  solutions,  and  opportunities  identified 
during  workshop  activities.  The  team’s  findings,  discussed  in  the  next  section,  were  based  on 
the  team’s  evaluation  of  the  answers  to  these  questions  against  (1)  results  of  the  two  working 
groups  and  descriptions  of  problems  and  (2)  potential  solutions  submitted  by  attendees  in  prep¬ 
aration  for  the  workshop.  The  IDA  study  team  also  furnished  observations  on  the  workshop 
activities  from  the  perspectives  of  the  team  members,  based  on  their  individual  experiences  and 
knowledge. 

Findings 

Are  there  critical  test  and  diagnostics  issues  or  problems  limiting  legacy  system  diag¬ 
nostic  performance  and  the  ability  to  meet  new  and future  diagnostic  demands?  Diagnostic  per¬ 
formance  of  defense  systems  is  not  commensurate  with  state-of-the-art  attainable  performance. 
Performance  limitations  constitute  critical  problems  that  result  in  increased  life  cycle  costs, 
decreased  systems  availability,  and  increased  support  and  maintenance  burdens. 

Are  the  pervasive  test  and  diagnostics  issues  differentiated  solely  by  the  legacy  systems 
diagnostic  performance  issues,  or  are  they  also  applicable  to  the  new  and  future  system  issues 
that  cross-cut  domains?  Issues  surrounding  the  test  and  diagnostics  problems  appear  pervasive 
and  similar  for  both  legacy  systems  and  systems  that  cut  across  domains. 

What  are  the  underlying  thrusts  or  new  directions  that  synthesize  proposed  solutions  of 
the pre -workshop  and  workshop  working  groups?  Thrusts  underlying  proposed  workshop  solu¬ 
tions  tend  toward  two  principal  areas:  increasing  awareness  of  integrated  diagnostics  benefits 
and  increasing  data  accuracy. 

What  are  the  technology  issues  needing  attention  to  achieve  proposed  solutions  and 
maximize  the  use  of  existing  technology  opportunities?  Potential  integrated  diagnostics  solu¬ 
tions  are  not  limited  by  current,  state-of-the-art,  nor  off-the-shelf  technologies. 

Is  infrastructure  of  integrated  diagnostics  a  key  element  in  proposed  solutions?  The 
most  significant  condition  limiting  improvements  is  an  infrastructure  that  lacks  of  an  open  or 
common  architecmre. 


Observations 

Lack  of  definition.  The  study  team  observed  that  although  the  workshop  participants 
recognized  significant  benefits  of  integrated  diagnostics,  they  lacked  a  clear  consensus  as  to  its 
definition.  In  the  opinion  of  the  IDA  study  team,  an  open  system  architecture  approach  for  inte¬ 
grated  diagnostics  would  resolve  many  of  the  definitional  concerns  cited  at  this  workshop. 

Open  system  architecture.  Integrated  diagnostics  implementations  are  best  achieved  by 
reducing  the  use  of  diagnostic  element  interfaces  that  are  implementation  unique  and  increas¬ 
ing  the  use  of  interface  standards  that  are  open,  well  defined,  non-proprietary,  and  commonly 
accepted.  An  open  system  architecture  would  give  the  Department  of  Defense  (DOD)  oppor¬ 
tunities  for  competitive  solutions,  reduced  implementation  costs,  improved  system  design  and 
support  flexibility,  and  incremental  technology  improvements. 

Data  as  diagnostics  information.  For  integrated  diagnostics  to  be  beneficial  and  effec¬ 
tive,  data  must  be  turned  into  information  that  is  accurate,  timely,  reliable,  and,  most  important¬ 
ly,  useful  in  helping  to  predict  and  eliminate  or  reduce  field  repair  requirements.  The  study  team 
recognized  a  potential  for  open  architectures  to  improve  information  usefulness  by  applying 
modularized  diagnostic  elements,  enhancing  timely  data  exchange,  and  adding  flexible  analy¬ 
sis  capabilities. 

Recommendation 

The  underlying  problem  of  all  the  individual  problems  discussed  by  the  participants 
was  the  lack  of  a  well-defined  framework  for  all  the  diagnostics  problems.  Establishing  an  open 
system  architecture  was  the  one  improvement  initiative  with  the  most  overall  benefits.  The 
team,  therefore,  recommended  that  DOD  should  conduct  a  two-phase  study  and  demonstration 
activity  to  estabhsh  an  integrated  diagnostics  open  system  architecture. 
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CHAPTER  1.  INTRODUCTION 


1.1  Purpose 

The  Institute  for  Defense  Analyses  (IDA)  conducted  a  Joint  Service  Integrated  Diag¬ 
nostics  Workshop  on  August  8, 1996,  under  the  auspices  of  the  Office  of  the  Director,  Industrial 
Capabilities  and  Assessments,  Under  Secretary  of  Defense  for  Acquisition  and  Technology. 
Participants  were  from  government  organizations  and  included  representatives  from  the  tech¬ 
nology  development,  acquisition,  and  support  functional  areas  of  the  Services  (Army,  Air 
Force,  Navy,  and  Marines).  The  initial  objectives  of  the  workshop  were  threefold: 

•  Increase  the  awareness  of  the  benefits  of  integrated  diagnostics  applications. 

•  Identify  weapon  system  support  problems  where  a  better  integrated  diagnostic 
approach  can  be  applied. 

•  Propose  integrated  diagnostics  opportunities  that  cross-cut  domains.^ 

During  the  course  of  the  workshop,  a  fourth  objective  was  identified:  to  begin  discus¬ 
sions  on  the  opportunities  of  applying  open  or  non-proprietary  architectures  to  integrated  diag¬ 
nostics.  Participants  in  this  workshop  were  divided  into  two  working  groups.  The  Legacy 
Systems  Working  Group  was  asked  to  identify  legacy  system  test  and  diagnostics  problems  and 
potential  solutions.  The  Cross-Cutting  Working  Group  was  asked  to  identify  test  and  diagnos¬ 
tics  problems  that  crossed  domains  as  well  as  potential  solutions  to  these  problems.  Both  work¬ 
ing  groups  focused  on  achieving  all  four  of  the  objectives. 

The  purpose  of  this  paper  is  to  document  workshop  activity  results  and  present  an  IDA 
study  team’s  analyses  of  the  workshop  results. 


'  In  the  context  of  the  workshop,  the  term  domain  was  intended  to  address  weapon  systems  and  their  capabilities 
that  span  from  legacy  systems  to  new  and  proposed  weapon  system  designs,  and  included  all  inter-Service 
weapon  system  applications  across  operational  boundaries  such  as  those  found  in  air,  land,  and  sea  system 
applications. 


1.2  Background 


On  October  11,  1988,  the  National  Security  Industrial  Association  (NSIA)  and  repre¬ 
sentatives  from  the  Office  of  the  Secretary  of  Defense  sponsored  executive  round-table  meet¬ 
ings  [NSIA  89]  on  implementing  integrated  diagnostics.  Examples  of  integrated  diagnostic 
elements  included  built-in  test,  automatic  test  systems,  and  data  collection  and  analysis  sys¬ 
tems.  The  NSIA  suggested  that  the  Department  of  Defense  (DOD)  should  improve  mainte¬ 
nance  through  diagnostics-development  discipline  and  integrating  diagnostics  elements.  The 
results  of  these  meetings  were  presented  by  the  NSIA  to  the  Under  Secretary  of  Defense  for 
Acquisition  on  April  14,  1989.  Various  studies  and  initiatives  since  that  time  have  clearly 
shown  that  total  diagnostic  performance  is  enhanced  by  the  synergistic  interaction  of  diagnostic 
elements  [Brown  90a,  90b;  TRW  96a,  96b]. 

There  are  several  descriptions  and  definitions  of  integrated  diagnostics  in  use  by  NSIA, 
the  Institute  of  Electrical  and  Electronics  Engineers  (IEEE),  and  various  Services  and  govern¬ 
ment  agencies  in  the  defense  community.  The  IDA  study  team  found  that  the  most  comprehen¬ 
sive  definition  was  put  forward  by  William  Keiner  [Keiner  90]  who  defined  integrated 
diagnostics  as  a  structured  process  which  maximizes  the  effectiveness  of  diagnostics  by 
integrating  the  individual  diagnostic  elements  of  testability,  automatic  testing,  manual  testing, 
training,  maintenance  aiding,  and  technical  information.” 

The  development  of  diagnostic  elements  takes  a  long  time  for  new  weapon  systems  and 
occurs  near  the  end  of  the  weapon  development  period.  While  there  have  been  some  improve¬ 
ments,  extensive  integration  of  diagnostic  elements  into  systems  once  fielded,  or  as  they 
become  legacy  systems,  has  often  been  very  difficult  to  justify.  Rarely  are  diagnostic  improve¬ 
ments  justified  as  stand-alone  initiatives.  Instead,  legacy  system  diagnostic  improvements, 
when  they  do  occur,  tend  to  be  secondary  benefits  of  system  modifications  justified  on  the  basis 
of  performance  enhancements.  The  degree  and  types  of  diagnostic  element  integration  are 
severely  limited  by  infrastructures  already  put  in  place  for  legacy  systems. 

Integrated  diagnostics  also  provides  an  effective  approach  for  the  prediction,  detection, 
and  isolation  of  faulty  conditions.  Integrated  diagnostics  helps  compensate  for  diagnostic  dif¬ 
ficulties  associated  with  increasing  subsystem  complexity  and  interdependency.  The  conse¬ 
quences  of  poor  diagnostic  performance  can  be  very  costly,  and  inaccurate  diagnosis  can  lead 
to  mission  failures,  reduced  readiness,  and  low  system  availability.  Examples  of  these  problems 
included  (1)  fault  isolation  ambiguities  resulting  in  excess  subsystem  removals,  (2)  a  higher- 
than-required  maintenance  and  repair  work  load,  (3)  an  increased  demand  in  spare  parts,  and 
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(4)  a  greater  “logistics  tail”^  in  terms  of  increases  in  support  and  test  equipment  requirements, 
support  and  maintenance  people,  and  transportation  needs. 

Integrated  diagnostics  represents  a  systems  approach  to  investigating  the  nature  or 
cause  of  hardware  and  software  anomalies  inhibiting  normal  operation,  a  necessary  step  in  the 
maintenance  and  repair  process.  Logistics  capabilities  are  enhanced  by  integrating  diagnostic 
elements.  Integrating  diagnostic  elements  creates  a  total  diagnostic  capability  that  exceeds  the 
upper  performance  limits  of  individual  support  and  maintenance  tools  operating  alone.  While 
the  specific  benefits  of  robust  integrated  diagnostics  capabilities  will  vaiy  by  application, 
reported  benefits  include  the  following: 

•  Greater  operational  and  mission  re^iiness 

•  Improved  systems  confidence 

•  Improved  systems  availability 

•  Reduced  depot  and  organizational  maintenance  work  loads 

•  Reduced  life-cycle  costs 

13  Approach 

In  preparation  for  the  workshop,  attendees  were  asked  to  submit  descriptions  of  test  and 
diagnostics  problems  as  well  as  proposed  solutions  to  these  problems.  In  the  working  group 
sessions,  participants  also  identified  test  and  diagnostics  problems  and  potential  solutions.  The 
IDA  study  team’s  first  step  was  to  consolidate  the  lists  of  problems  and  solutions  into  a  set  of 
tables  for  easier  reference  and  analyses.  The  results  of  the  workshop,  as  summarized  in  these 
tables,  represented  the  open  dialogue  over  veiy  broad  topic  areas  and  the  viewpoints  of  other 
Service  and  DOD  personnel  from  disparate  organizational  and  functional  assignments. 

Given  the  range  of  topic  areas  and  viewpoints,  the  IDA  study  team  set  out  to  identify 
the  common  threads  or  issues  found  in  these  tables.  The  IDA  approach  to  the  analyses  of  the 
workshop  minutes  and  discussion  was  to  provide  a  general  assessment  of  issues  and  opportu¬ 
nities  tending  to  be  pervasive  throughout  the  lists  of  problems  and  solutions.  The  IDA  study 
team  developed  and  addressed  the  following  questions: 

•  Are  there  critical  test  and  diagnostics  issues  or  problems  limiting  legacy  system 
diagnostic  performance  and  the  ability  to  meet  new  and  future  diagnostic  demands? 

^  The  “logistics  tail”  is  the  chain  of  logistics  support  that  goes  from  the  battlefield  back  to  the  continental  United 
States  maintenance  depots,  factories,  and  contractor  support  personnel. 


•  Are  the  pervasive  test  and  diagnostics  issues  differentiated  solely  by  the  legacy  sys¬ 
tems  diagnostic  performance  issues,  or  are  they  also  applicable  to  the  new  and 
future  system  issues  that  cross-cut  domains? 

•  What  are  the  underlying  thrusts  or  new  directions  that  synthesize  proposed  solu¬ 
tions  of  the  pre-workshop  and  workshop  working  groups? 

•  What  are  the  technology  issues  needing  attention  to  achieve  proposed  solutions  and 
maximize  the  use  of  existing  technology  opportunities? 

•  Is  infrastructure  of  integrated  diagnostics  a  key  element  in  proposed  solutions? 

The  first  three  questions  were  designed  to  look  for  issues  and  opportunities  that 
appeared  to  be  critical,  pervasive,  and  underlying  in  nature.  The  fourth  question,  based  on  the 
major  role  that  technology  evolution  and  advances  have  played  in  defense  systems  develop¬ 
ment,  addressed  whether  technology  might  also  play  a  role  in  solving  perceived  problems.  The 
fifth  question  explored  what  role  infi-astructure  appears  to  play  in  integrating  diagnostic  ele¬ 
ments  across  the  proposed  solutions.  The  answers  to  these  questions,  based  on  the  problems 
and  solutions  resulting  from  the  workshop  activities,  appear  in  Chapter  3. 

1.4  Organization 

Chapter  2  contains  summary  tables  and  discussions  of  workshop  activities.  Created  by 
the  IDA  study  team,  these  tables  were  used  by  the  study  team  as  the  basis  for  the  team’s  find¬ 
ings. 

Chapter  3  contains  the  IDA  study  team’s  analyses  of  the  discussions,  problems,  and 
solutions  raised  by  the  two  working  groups  in  separate  and  combined  sessions.  The  analyses 
are  organized  around  the  five  questions  identified  previously  in  Section  1.3.  Chapter  3  also 
includes  the  IDA  study  team’s  own  observations  of  the  workshop.  The  team’s  original  intent 
was  to  isolate  critical  integrated  diagnostics  issues  and  conditions  influencing  military  systems 
acquisition,  maintenance,  support,  and  repair,  and  to  coordinate  the  observations  with  all  work¬ 
shop  participants  and  their  respective  organizations  about  these  issues  and  conditions.  This  was 
not  feasible  because  of  time  constraints;  therefore,  these  observations  represent  only  the  IDA 
study  team’s  perspective  and  may  not  reflect  a  DOD-wide  position.  Finally,  the  EDA  study  team 
offers  a  single  recommendation  based  on  the  key  integrated  diagnostics  improvements  or  activ¬ 
ities  that  would  have  the  greatest  potential  of  broadly  enhancing  DoD  weapons  systems  main¬ 
tenance  and  support  capabilities. 
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Appendix  A  contains  the  minutes^  of  the  sessions  of  the  two  working  groups,  focusing 
on  test  and  diagnostics  problems  in  legacy  systems  and  those  problems  that  cut  across  domains, 
as  well  as  possible  solutions.  Appendices  B  through  H  are  reproductions  of  the  briefings  given 
at  the  workshop.  A  list  of  references  and  a  list  of  acronyms  are  provided  at  the  end  of  the  doc¬ 
ument. 
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The  minutes  have  been  approved  by  the  chairs  of  the  two  working  groups  and  distributed  to  workshop  partici¬ 
pants  prior  to  the  publication  of  this  IDA  paper.  Therefore,  no  editorial  changes  have  been  made  to  Appendix  A. 


CHAPTER  2.  SUMMARY  OF  WORKSHOP  ACTIVITIES 


The  combination  of  activities  leading  up  to  the  1996  Joint  Service  Integrated  Diagnos¬ 
tics  Workshop  and  the  sessions  conducted  by  the  two  working  groups  provided  three  relatively 
independent  views  of  test  and  diagnostics  problems  and  potential  solutions.  The  problems  and 
solutions  are  summarized  in  tables  created  by  the  IDA  study  team  and  are  presented  in  the  fol¬ 
lowing  sections. 

2.1  Pre- Workshop  Descriptions  of  Integrated  Diagnostics  Problems  and  Solutions 

In  preparation  for  the  1996  Joint  Service  Integrated  Diagnostics  Workshop,  attendees 
were  asked  to  submit  descriptions  of  test  and  diagnostics  problems,  as  well  as  proposed  solu¬ 
tions  to  these  problems.  The  IDA  study  team  used  the  following  approach  to  organize  and  sum¬ 
marize  the  submitted  descriptions: 

•  General  categories  were  selected  to  match  descriptions. 

•  Ideas  and  concepts  were  reduced  to  general  thoughts  marked  by  bullets. 

•  Screening  of  details  was  limited  to  simplifying  presentations. 

•  No  weight  was  assigned  to  presentation  order. 

Seven  general  categories  were  selected  and  summary  descriptions  of  the  results  are  pre¬ 
sented  in  Table  1. 


Table  1.  Problems  and  Solutions  Submitted  Prior  to  Workshop 


General 

Category 

Problems 

Proposed  Solutions 

General 

Topics 

•  Global  (-illities,  manpower,  costs,  etc.) 

•  Confusion  about  integrated  diagnostics  process/ 
practices  (capabilities,  engineering,  variety  of 
testing  approaches) 

•  Tenuous  links  to  quality  &  process  improve¬ 
ments 

•  Education 

•  Exposure  to  successes 

•  Focus  on  total  quality  as  integral  element  of 
integrated  diagnostics 

•  New  integrated  diagnostics  process  guides/ 
manual 

Technology  Training  Exchange  Interfaces  &  Diagnostic  General 

Transfer  &  Presentation  Architectures  Performance  Category 


Table  1.  Problems  and  Solutions  Submitted  Prior  to  Workshop  (Continued) 


Problems 

Proposed  Solutions 

•  Global  (RTOK/CND,  excessive  maintenance, 
etc.) 

•  Insufficient  embedded  testing  capabilities 

•  Fault  isolation  ambiguities 

•  Poor  isolation/duplication  of  intermittent  faults 

•  Poor  software  anomaly  identification/isolation 

•  Limited  prognostic  capabilities 

•  Limited  analysis  capabilities  for  multiple  fail¬ 
ures 

•  Embedded  and  distributed  test  and  mainte¬ 
nance  buses 

•  Advanced  diagnostic  sensors 

•  Data  fusion  across  and  among  diagnostic  ele¬ 
ments 

•  Focus  on  development  tools  and  support 
environment 

•  Point  solutions,  inflexible  to  change  or  reuse 

•  Costly  to  redevelop  different  solution  to  same 
generic  problem 

•  No  consistent  data  types  nor  common  means  of 
accessing  data 

•  No  common  legacy  system  electronic  interfaces 

•  Proliferation  of  equipment,  aids,  tools,  etc. 

•  Provide  interface-based  requirements  for 
transparency  &  interchangeability 

•  Establish  data  access  standards  &  common 
approaches  (buses,  protocols,  formats,  etc.) 

•  Implement  open  architecture  for  all  diagnos¬ 
tic  elements 

•  Provide  remote  services  links  (“help  desks”) 

•  Develop  tool  suite  devoted  to  integrated 
diagnostics  tasks 

•  Data  security  &  integrity  (for  accident  &  threat) 

•  Accurate,  detailed,  real-time  configuration  sta¬ 
tus 

•  Electronic  display  limitations  (size,  “washout”®) 

•  Common/standard  approaches  for  automated 
data  capture 

•  Data  entry  limitations 

•  Self-checking  and  self-correcting  data  media 

•  Alternate  input/output  (video,  voice,  etc.) 

•  Capitalize  on  Internet  &  multimedia 

•  Exploit  electronic  memory  &  search  tech¬ 
niques 

•  Remote  data  access  for  configuration, 
update,  logs 

•  Seamless  communications  nets 

•  Growing  system  complexity 

•  Variety /proliferation  of  system  types  &  configu¬ 
rations 

•  Integrated  &  interdependent  systems 

•  Support  tool  complexity  (needs  own  training) 

•  Real-time  expert  guidance  (analysis  &  artifi¬ 
cial  intelligence) 

•  Remote  diagnostican  &  operator  support 

•  Easy-to-use  tools  &  aids 

•  Tools  that  leam  with  operators 

•  Principles  &  methods  not  totally  understood 

•  Available  technology  not  being  applied 

•  Improvements  not  reaching  legacy  systems 

•  Improvement  needs  not  substantiated,  support¬ 
ed,  or  funded 

•  Build  support  through  demonstrations 

Table  1.  Problems  and  Solutions  Submitted  Prior  to  Workshop  (Continued) 


General 

Category 

Problems 

Proposed  Solutions 

Business 

Practices 

•  Need  focal  points  for  integrated  diagnostics 

•  Services/programs  focus  on  individual  needs 
versus  common  benefits 

•  Contractor  interests  follow  funding  (infrequent¬ 
ly  aligns  with  integrated  diagnostics  goals) 

•  DOD  lags  commercial  technology/business 
practices 

•  Difficult  to  keep  up  with  all/new  solutions 

•  Establish  linked  integrated  diagnostics  offic¬ 
es  in  Services 

•  Establish  integrated  diagnostics  R&D  facili¬ 
ty  &  dedicated  dollars 

•  Establish  system  of  virtual  integrated  diag¬ 
nostics  offices  with  links  across  DOD  lines 

•  Conduct  demonstrations  (get  things  started) 

•  Establish  Joint  Office  on  Integrated  diagnos¬ 
tics  to  keep  up  with  technology 

•  Take  advantage  of  existing  capabilities 

a.  “Washout”  refers  to  contrast  problems  (e.g.,  too  bright,  glare)  on  the  display. 


2.2  Legacy  Systems  Working  Group 

In  the  context  of  the  Workshop,  the  term  legacy  was  used  to  represent  existing  systems. 
From  this  perspective,  the  Legacy  Systems  Working  Group  was  tasked  to  identify  legacy  sys¬ 
tem  test  and  diagnostics  problems  and  potential  solutions.  Table  2  summarizes  the  results  of 
this  working  group.  More  detailed  discussions  are  contained  in  the  Workshop  minutes  in 
Appendix  A. 


Table  2.  Legacy  Systems  -  Problems  and  Solutions 


Problems 

Proposed  Solutions 

Test  data  and  diagnostic  interfaces  are  not  standard  at 
any  level  (functional,  system,  platform,  or  service). 

—  No  consensus  solutions  identified  — 

Diagnostic  capability  is  hampered  by  poor  configura¬ 
tion  management  of  design  and  test  documentation 
over  life  cycle.® 

—  No  consensus  solutions  identified  — 

Non-standard  data  collection,  transfer,  and  storage, 
and  validity  of  operational,  maintenance,  &  configu¬ 
ration  data  are  questionable. 

—  No  consensus  solutions  identified  — 

Need  for  on-the-job  training  is  increasing,  while 
number  of  maintainers  and  amount  of  formal  training 
received  are  decreasing. 

Conduct  demonstrations  to  show  where  technology 
may  compensate  for  formal  classroom  training  defi¬ 
ciencies. 

Processes  for  linking  diagnostic  improvements, 
acquisition  streamlining,  and  commercial  off-the- 
shelf  products/parts  are  not  well  defrned. 

Recommended  processes  should  be  developed  and 
included  into  the  new  DOD  or  Service  acquisition 
deskbooks. 

Table  2.  Legacy  Systems  -  Problems  and  Solutions  (Continued) 


Problems 

Proposed  Solutions 

Diagnostic  improvements  are  not  occurring  because 
maintenance  equipment  capability  and  test  methods 
are  not  being  tracked  against  actual  field  procedures 
and  failures. 

Improve  data  collection  and  tracking  and  utilize  to 
implement  diagnostic  improvements; 

Policy  should  not  dictate  that  all  solutions  be  “gener¬ 
al”  or  “standard.”  Both  are  good  for  some  applica¬ 
tions. 

Capabilities  should  be  made  available  to  permit  dem¬ 
onstration  and  trading-off  of  general  versus  point 
solutions. 

An  integrated  diagnostics  clearing  house  does  not 
exist — ^integrated  diagnostics  information  sharing 
and  transfer  are  limited. 

A  Joint-Service  group  should  be  established  with 
responsibility  for  drafting  integrated  diagnostics  stra¬ 
tegic  plan. 

a.  Problems  frequently  start  in  acquisition  when  capabilities  are  not  bought  or  else  not  bought  in  usable 
format. 


2.3  Cross-Cutting  Working  Group 

The  Cross-Cutting  Working  Group  was  tasked  to  identify  test  and  diagnostics  problems 
that  cut  across  domains  as  well  as  potential  solutions  to  these  problems.  In  the  context  of  the 
workshop,  the  term  domain  was  intended  to  address  systems  and  their  capabilities  that  span 
from  legacy  systems  to  new  and  proposed  weapon  system  designs,  and  included  all  inter-Ser- 
vice  weapon  system  applications  across  operational  boundaries  such  as  those  found  in  air,  land, 
and  sea  system  applications.  Table  3  briefly  summarizes  the  Cross-Cutting  Working  Group 
results.  More  detailed  discussions  are  contained  in  the  workshop  minutes  in  Appendix  A. 


Table  3.  Cross-Cutting  Domains  -  Problems  and  Solutions 


Problems 

Proposed  Solutions 

Lack  of  common  integrated  diagnostics 

•  Definitions 

•  Functional  architectures  &  interfaces 

•  Measures  of  effectiveness 

Establish  a  U.S.  govemment/industry  consortium  to 
resolve. 

Lack  of  common  processes  for  instituting  integrated 
diagnostics  over  system  life  cycles 

—  No  consensus  solutions  identified  — 

Personal  and  organizational  objectives  can  reduce 
accuracy  of  field  maintenance  data. 

Remove  subjectivity  by  increasing  automated  report¬ 
ing. 

Uncommon  and  un-automated  data  collection 
schemes  impede  integration  of  diagnostic  informa¬ 
tion. 

—  No  consensus  solutions  identified  — 
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Table  3.  Cross-Cutting  Domains  -  Problems  and  Solutions  (Continued) 


Problems 

Proposed  Solutions 

Processes  for  integrating  diagnostic  data  and  capa¬ 
bilities  are  not  well  defined  nor  under  organizational 
control  or  sponsorship. 

—  No  consensus  solutions  identified  — 

Analog  built-in-test  (BID  capabilities  lack  common 
protocols,  standards,  or  techniques. 

Identify  BIT  formats  and  address  trade-offs  of  on¬ 
board  vs.  off-line  testing.® 

a.  Presented  as  an  unfunded  study  under  Task  #1 1  of  the  DOD  ATS  Executive  Agent  R&D  Project,  and 
addressed  in  more  detail  in  Appendix  H  of  this  paper. 


CHAPTERS.  ANALYSES 


The  analyses  in  this  chapter  are  based  on  the  IDA  study  team’s  assessment  of  the 
answers  to  the  five  questions  identified  in  the  approach  section  of  Chapter  1.  Each  question  is 
addressed  in  Section  3.1,  with  the  finding  followed  by  the  IDA  study  team’s  discussion.  Section 
3.2  presents  the  IDA  study  team’s  observations  of  the  workshop,  and  a  recommendation  is  giv¬ 
en  in  Section  3.3. 

3.1  Findings 

Are  there  critical  test  and  diagnostics  issues  or  problems  limiting  legacy  system  diagnos¬ 
tic  performance  and  the  ability  to  meet  new  and  future  diagnostic  demands? 

Finding  1.  Diagnostic  performance  of  defense  systems  is  not  commensurate 
with  state-of-the-art  attainable  performance.  The  resulting  performance  limita¬ 
tions  constitute  critical  problems  causing  increased  life  cycle  costs,  decreased 
systems  availability,  and  increased  support  and  maintenance  burdens. 

The  IDA  study  team  defined  a  test  or  diagnostics  problem  as  a  condition  that  limited 
weapon  system  capability  or  performance.  For  the  purpose  of  this  analysis,  problems  were  con¬ 
sidered  critical  when  current  practices  for  identifying  and  correcting  a  problem  were  more  cost¬ 
ly  than  alternative  practices  benefiting  from  the  integration  of  diagnostic  elements. 

Evidence  that  critical  test  and  diagnostic  limitations  exist  was  provided  by  the  vast 
range  and  large  numbers  of  problems  presented  by  the  workshop  attendees  (summarized  in 
Tables  1, 2,  and  3  in  Chapter  2). 

However,  verifying  that  a  limiting  condition  constitutes  a  critical  problem  is  much  hard¬ 
er  to  accomplish  with  certainty.  Anecdotal  evidence  of  cost  benefits  derived  from  specific  inte¬ 
grated  diagnostics  improvements  was  presented  both  during  the  introductoiy  briefings  and  by 
attendees  during  the  working  group  discussions.  The  source  of  benefits  cited  most  frequently 
tended  to  be  the  elimination  of  duplicative  efforts  (e.g.,  re-developing,  re-testing,  re-working). 
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The  IDA  study  team  observed  that  over  the  past  years  there  have  been  no  other  suitable 
measurement  mechanisms  introduced,  short  of  conducting  system-level  integrated  diagnostic 
demonstration  programs.  Demonstration  programs  were  cited  as  tools  that  may  be  used  by 
DOD  to  verify  the  specific  benefits  of  adopting  integrated  diagnostics  improvements  in  both 
legacy  and  future  weapon  systems. 

The  IDA  study  team  found  that  critical  test  and  diagnostics  problems  diminished  weap¬ 
on  system  capabilities  or  performance  and  increased  the  need  for  additional  off-line  testing, 
formal  and  on-the-job  personnel  training,  the  necessary  development  of  additional  technical 
information,  etc.  This  finding  was  based  on  (1)  the  compelling  arguments  presented  by  attend¬ 
ees,  (2)  the  sheer  number  of  similar  conditions  that  might  benefit  from  common  actions,  and 
(3)  the  anecdotal  evidence  of  cost  benefits  (in  terms  of  both  dollars  and  resources)  realized  in 
past  integrated  diagnostic  demonstrations  [MDA  96].  These  past  demonstrations  were  initiated 
by  DOD  to  increase  aircraft  availability  by  improving  diagnostic  accuracy  while  reducing 
maintenance  man-hours. 

Are  the  pervasive  test  and  diagnostics  issues  differentiated  solely  by  the  legacy  systems 
diagnostic  performance  issues,  or  are  they  also  applicable  to  the  new  and  future  system 
issues  that  cross-cut  domains? 

Finding  2.  The  pervasive  mix  of  test  and  diagnostics  issues  tends  to  be  similar 

for  legacy  systems  and  for  systems  that  cross-cut  domains. 

For  this  part  of  the  analyses,  the  IDA  study  team  looked  at  the  similarities  and  differ¬ 
ences  among  the  issues  identified  during  the  sessions  of  the  Legacy  Systems  and  Cross-Cutting 
Working  Groups. 

The  IDA  study  team  chose  to  look  for  issues  associated  with  the  summarized  problems 
and  potential  solutions  presented  in  both  Tables  2  and  3  in  Chapter  2.  This  approach  was  select¬ 
ed  because  of  the  desire  to  assess  issues  that  were  “pervasive”  across  different  applications. 
Then,  recognizing  a  degree  of  overlap  between  summaries  presented  in  the  individual  problem 
categories,  the  IDA  study  team  drafted  the  following  set  of  test  and  diagnostics  issues.  Once 
these  issues  were  identified,  it  permitted  formulation  of  a  recommended  architecture  approach 
on  how  DOD  can  contain  costs  for  both  its  legacy  systems  and  new  systems  and  how  to 
improve  on  their  diagnostic  performance  capabilities. 

a.  Paucity  of  common  data  and  diagnostic  system  interfaces.  The  lack  of  common 
interfaces,  protocols,  data  structures,  etc.,  was  cited  throughout  by  both  working 
groups.  This  issue  was  also  discussed  openly  during  the  workshop  sessions  and  reg- 


ularly  during  introductory  briefings.  For  example,  the  cross-cutting  problems  and 
solutions  of  the  issue  were  highlighted  as  data  definition  differences  between  the 
Services  on  the  Joint  Advanced  Strike  Technology  (JAST)  program.  In  a  similar 
vein,  the  legacy  system  nature  of  the  issue  was  highlighted  as  data  formats  and  pro¬ 
tocols  on  the  Integrated  Maintenance  Data  System  (IMDS). 

b.  Quality  and  validity  of  design,  documentation,  and  configuration  data.  This  issue 
was  only  cited  in  the  Legacy  Systems  Working  Group  sessions.  The  IDA  study  team 
noted  that  a  possible  reason  for  this  difference  may  be  newer  systems  are  doing  a 
better  job  of  acquiring  and  maintaining  essential  diagnostics-related  documenta¬ 
tion.  Another  reason  may  be  that  similar  problems  have  not  yet  been  identified  on 
newer  systems  (e.g.,  not  yet  fielded).  A  third  reason  may  be  that  new  systems  have 
an  evolving  infrastructure  that  may  more  readily  accommodate  and  integrate  diag¬ 
nostic  elements  than  does  an  established  legacy  infrastructure.  However,  there  was 
insufficient  information  to  characterize  why  this  difference  was  only  noted  in  the 
Legacy  Systems  Working  Group. 

c.  Non-standard  data  collection,  tranter,  storage,  and  meaning.  This  issue  was  con¬ 
sistently  cited  by  both  working  groups.  No  significant  differences  were  observed  by 
the  IDA  study  team  members.  Similar  to  the  paucity  of  common  data  and  diagnostic 
system  interfaces,  this  issue  was  cited  in  the  JAST  and  IMDS  introductory  brief¬ 
ings. 

d.  Training.  This  issue  was  only  cited  in  the  Legacy  Systems  Working  Group  sessions. 
The  IDA  study  team  observed  that  a  possible  reason  for  this  difference  may  be  that 
newer  systems  are  doing  a  better  job  of  initial  training.  Another  reason  may  be  due 
to  the  fact  that  similar  problems  have  not  yet  been  identified  on  newer  systems  (e.g., 
not  yet  fielded).  A  third  reason  may  be  that  newer  systems  have  achieved  higher 
levels  of  diagnostics  element  integration,  and  additional  training  for  legacy  systems 
is  required  to  make  up  for  the  lack  of  integrated  diagnostics  tools  and  the  informa¬ 
tion  exchange  across  diagnostic  elements.  However,  there  was  insufficient  informa¬ 
tion  to  characterize  why  this  difference  was  only  noted  in  the  Legacy  Systems 
Working  Group. 

e.  Need  for  centralized focus  and  better  process  definition.  This  issue  was  consistently 
cited  by  both  working  groups.  No  significant  differences  were  observed  by  the  IDA 
study  team  members. 


f.  Accuracy  of  data  collection  and  reporting.  This  issue  was  consistently  cited  by  both 
working  groups.  No  significant  differences  were  observed  by  the  IDA  study  team 
members. 

Two  of  the  six  test  and  diagnostics  issues  were  not  identified  as  issues  in  the  Cross-Cut¬ 
ting  Working  Group:  item  b.  Quality  and  validity  of  design,  documentation,  and  configuration 
data;  and  item  d.  Training.  The  other  four  issues  were  identified  by  both  working  groups,  and 
these  issues,  as  stated  in  the  discussions  in  both  groups,  were  nearly  identical  in  nature.  This 
led  to  the  IDA  study  team’s  finding  that  the  pervasive  issues  tend  to  be  similar  for  legacy  sys¬ 
tems  and  for  systems  that  cross-cut  domains. 

What  are  the  underlying  thrusts  or  new  directions  that  synthesize  proposed  solutions  of 
the  pre-workshop  and  workshop  working  groups? 

Finding  3.  The  identified  solutions  chiefly  addressed  two  principal  areas: 

•  Increasing  awareness  of  integrated  diagnostics  benefits.  This  solution 
would  increase  the  awareness  of  integrated  diagnostics  benefits  through  in¬ 
tegrated  diagnostics  demonstrations,  groups/centers  of  integrated  diagnos¬ 
tics  focus,  integrated  diagnostics  process  guides,  global  integrated 
diagnostics  plans,  formal  training,  etc.,  all  with  the  same  general  focus  to¬ 
wards  consistent  integrated  diagnostics  definitions,  standards,  and  measures 
of  effectiveness. 

•  Increasing  data  accuracy.  This  solution  would  verify  integrated  diagnostics 
benefits  and  justify  improvements  The  automation  of  some  maintenance 
data  collection  would  be  a  major  step  to  improving  accuracy  by  removing 
subjectivity  and  enhancing  data  exchange. 

For  this  part  of  the  analyses,  the  IDA  study  team  characterized  the  proposed  solutions 
presented  by  the  working  groups.  The  IDA  study  team  looked  for  common  links  among  the 
rationales  behind  each  proposed  solution.  Two  common  links  appeared  prominent:  increasing 
awareness  of  integrated  diagnostics  and  increasing  data  accuracy.  The  approaches  to  accom¬ 
plishing  the  goal  of  extending  or  increasing  awareness  of  integrated  diagnostics  benefits 
included  (1)  the  establishment  of  groups  or  forums  with  responsibilities  for  identifying  future 
directions,  and  (2)  the  use  of  demonstrations  to  verify  the  benefits  of  improved  diagnostics 
capabilities. 


From  the  data  accuracy  perspective,  the  rationale  behind  the  proposed  solutions  came 
from  three  different  sources:  the  pre- workshop  and  mail-in  activity,  and  the  two  independent 
working  groups  assembled  during  the  workshop.  One  solution  was  to  use  better  information, 
resulting  from  improved  data  accuracy,  to  verify  integrated  diagnostics  benefits  (this  focus  is 
not  totally  independent  of  the  first  common  link:  increasing  awareness  of  integrated  diagnos¬ 
tics).  Another  solution  was  to  directly  address  a  major  contributor  to  data  accuracy  problem  by 
removing  individual  and  organizational  subjectivity  through  increased  automation  of  data  col¬ 
lection.  The  third  solution  was  to  transform  raw  data  into  timely  information  through  integrated 
diagnostics  synthesis  tools,  thus  providing  the  right  information  at  the  right  time. 

The  IDA  study  team  noted  the  absence  of  proposed  solutions  that  would  have  addressed 
the  lack  of  common  interfaces,  non-standard  data  collection  and  transfer  schemes,  and  the  lack 
of  common  processes  for  instituting  integrated  diagnostics.  The  IDA  study  team  believes  that 
these  problem  areas  fall  under  the  general  category  of  architectures  and  interfaces,  which  are 
discussed  in  more  detail  in  the  discussion  under  Finding  5. 

What  are  the  technology  issues  needing  attention  to  achieve  proposed  solutions  and  max¬ 
imize  the  use  of  existing  technology  opportunities? 

Finding  4.  Integrated  diagnostics  improvements  are  not  limited  by  current, 

state-of-the-art,  nor  off-the-shelf  technologies. 

For  this  part  of  the  analyses,  the  IDA  study  team  set  out  to  identify  any  new  technolo¬ 
gies  that  may  be  needed  to  achieve  the  proposed  solutions  in  Tables  1,  2,  and  3  in  Chapter  2. 
The  IDA  study  team  reviewed  all  of  the  proposed  solutions,  and  made  an  effort  to  assess  wheth¬ 
er  limitations  posed  by  currently  available  technology  would  inhibit  solution  implementation. 
With  the  exception  of  two  specific  categories*  of  existing  and  proven  technologies  discussed 
in  the  workshop  minutes  (Appendix  A),  the  IDA  study  team  did  not  believe  there  were  any 
immediate  limitations  to  implementing  improved  diagnostics  capabilities  imposed  by  current 
or  available  technologies. 

The  study  team  found  that  opportunities  for  improving  integrated  diagnostics,  identi¬ 
fied  during  the  workshop  activities,  can  be  realized  by  applying  current,  state-of-the-art,  or  off- 
the-shelf  technologies.  In  general,  the  IDA  study  team  agreed  with  the  statement  put  forward 
during  the  Cross-Cutting  Working  Group  sessions  that  “Integrated  diagnostics  is  not  a  technical 
problem — ^it  is  a  political,  cultural,  organizational  problem.”  However,  the  IDA  study  team’s 


Analog  built-in  test  (BIT)  and  advanced  diagnostic  sensors. 
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opinion  is  that  future  technology  advances  will  provide  further  opportunities  to  enhance  diag¬ 
nostic  accuracy,  performance,  and  cross-cutting  capabilities. 

Is  infrastructure  of  integrated  diagnostics  a  key  element  in  proposed  solutions? 

Finding  5.  The  most  significant  condition  limiting  integrated  diagnostics  im¬ 
provements  is  the  lack  of  an  open  system  architecture. 

For  this  part  of  the  analyses,  the  IDA  study  team  reviewed  and  categorized  the  prob¬ 
lems,  submitted  by  both  working  groups,  that  were  based  on  system  capability  to  exchange 
information  across  interfaces. 

The  IDA  study  team  recognized  that  the  term  infrastructure  could  have  a  variety  of 
interpretations.  In  the  context  of  integrated  diagnostics,  the  IDA  study  team  defined  infrastruc¬ 
ture  to  represent  a  collection  of  hardware  and  software  elements,  interfaces,  policies,  and  pro¬ 
cesses  that  provide  the  means  of  implementing  a  support  capability.  In  an  effort  to  further  scope 
the  analysis,  the  IDA  study  team  looked  at  characteristics  of  open  system  architectural  inter¬ 
faces  for  diagnostic  elements,  with  specific  emphasis  on  the  exchange  and  use  of  data  needed 
to  support  testing,  diagnostics,  and  maintenance.  The  IDA  study  team  then  identified  problem 
areas  that  focused  on  difficulties  and  limitations  posed  by  data  exchange  practices  between  and 
across  interfaces. 

The  type  and  nature  of  identified  interface  difficulties  and  limitations  cited  in  both  the 
introductory  presentations  and  during  working  group  sessions  are  listed  in  the  following  para¬ 
graphs.  (Note  that  the  pervasive  nature  of  these  difficulties  and  limitations  was  already  dis¬ 
cussed  previously  in  the  Finding  2  discussion,  items  a,  c,  and/.) 

a.  Proliferation  of  data  collection  schemes  and  media.  There  are  no  common  methods, 
equipment,  or  practices  for  diagnostic  element  data  capture  across  system  and  ser¬ 
vice  applications. 

b.  Unique  diagnostic  element  interfaces.  There  is  a  lack  of  common  data  interface 
links  (hardware  or  software)  between  diagnostic  element  applications,  leading  to  a 
proliferation  of  system  unique  solutions. 

1.  Electrical  interface  characteristics  (geometry,  pin-outs,  power,  etc.)  are  unique 
to  most  weapons  systems  applications,  and  to  most  subsystems  within  these 
applications. 

2.  Information  protocols  for  accessing,  transmitting,  or  servicing  data  across  inter¬ 
faces  are  unique  for  most  applications. 


3.  Data  formats  and  data  communication  approaches  are  unique  for  most  applica¬ 
tions. 

c.  Limited  data  access.  Data  storage  schemes  and  database  architectures  are  not 
responsive  to  integrated  diagnostics  needs  because  of  slow  and  ineffective  queiy 
capabilities,  data  ambiguities,  and  inabilities  to  link  interdependent  data  records 
within  and  across  databases. 

d.  Inconsistent  data  types  and  definitions.  Data  type,  name,  and  definition  variations 
by  services,  diagnostic  elements,  and  weapon  system  applications  limit  commonal¬ 
ity  of  diagnostic  tools  and  inhibit  information  sharing. 

e.  Limited  common  embedded  and/or  distributed  test  and  maintenance  busses.  Phys¬ 
ical,  functional,  and  protocol  accesses  to  on-board  system  test  and  maintenance  data 
interfaces  tend  to  vary  by  specific  weapon  system,  the  subsystems  installed  in  the 
weapon  system,  service,  diagnostic  elements,  and  product  age. 

f.  Limited  universal  data  fusion  and  analysis  schemes.  Diagnostic  information  pro¬ 
cessing  and  sharing  schemes  are  in  limited  use,  and  prototype  tool  implementations 
(such  as  knowledge-based  systems  or  model-based  reasoners)  tend  to  be  limited  in 
design  to  a  unique  product. 

The  proliferation  of  unique  diagnostic  interfaces,  as  noted  in  the  workshop  and  summa¬ 
rized  in  this  list,  was  attributed  to  the  following  list  of  factors  identified  during  the  workshop. 
The  consensus  of  the  IDA  study  team  was  that  these  factors  play  a  major  role  in  perpetuating 
the  presence  of  product-unique  and  sometimes  proprietary  integrated  diagnostics  interfaces: 

•  The  lack  of  agreed-to  standards  for  common  and  reusable  applications. 

•  The  use  of  system-specific  diagnostic  elements  with  their  own  unique  and/or  pro¬ 
prietary  interfaces. 

•  The  lack  of  near-term  incentives  for  a  contractor  or  weapon  system  program  office 
to  compensate  for  increased  project  cost  and  risks  by  addressing  common  needs 
across  applications.^ 


While  the  working  groups  felt  long-term  life-cycle  costs  are  typically  reduced  when  logistics  capabilities  are 
enhanced  by  the  interaction  of  individual  diagnostic  elements  as  a  systems  approach,  most  near-term  imple¬ 
mentation  costs  are  expected  to  be  higher,  especially  for  the  first-time  development  and  implementation  of  a 
new  common  capability. 


The  consequences  of  unique  and  limiting  interface  conditions  included  limited  data 
sharing,  limited  tool  interchangeability,  proliferation  of  unique  and  non-reusable  diagnostic 
tools,  and  proprietary  or  military  design-specific  solutions.  Therefore,  open  system  architecture 
interfaces  for  integrated  diagnostics  elements  were  considered  essential  to  an  infrastructure 
characterized  by  growth  and  evolution  flexibility,  reusability  potential,  and  high  effectiveness 
attributes.  Based  on  this  short  analysis,  the  study  team  found  the  lack  of  an  open  system  archi¬ 
tecture  that  includes  industry  standards  body  or  commonly  accepted  interface  specifications 
and  standards  at  diagnostic  element  interfaces  to  be  most  significant  condition  limiting  inte¬ 
grated  diagnostics  improvements. 

3.2  Observations 

This  section  presents  observations  on  the  workshop  processes  and  results  from  the  per¬ 
spective  of  the  IDA  study  team.  The  original  intent  was  to  isolate  critical  integrated  diagnostics 
issues  and  conditions  influencing  military  systems  acquisition,  maintenance,  support,  and 
repair.  As  it  was  not  feasible  to  coordinate  with  all  workshop  participants  and  their  respective 
organizations  under  given  time  constraints,  the  observations  presented  herein  represent  the 
IDA  study  team’s  perspective,  based  on  its  experiences  and  knowledge,  and  may  not  reflect  a 
DOD-wide  interpretation. 

Definition  of  integrated  diagnostics 

Observation  1.  While  the  workshop  participants  recognized  the  significant 
benefits  of  integrating  diagnostics,  they  lacked  a  clear  consensus  as  to  a  defini¬ 
tion  of  integrated  diagnostics. 

From  the  pre-workshop  problem  and  solution  descriptions  and  throughout  the  work¬ 
shop  proceedings,  participants  cited  the  need  for  a  clear  definition  of  integrated  diagnostics. 
Their  concerns  were  markedly  similar  to  those  identified  previously  at  an  Integrated  Diagnos¬ 
tics  Workshop  conducted  by  IDA  on  June  21  and  22, 1989,  and  August  3, 1989.  Differences  in 
understanding  and  definitions  were  identified  when  defining  “integrated  diagnostics”  at  the 
1989  workshop,  as  shown  in  the  following  quotation,  and  they  remain  unresolved  at  the  more 
recent  workshop  held  in  1996. 

For  example,  the  term  “Integrated  Diagnostics”  was  used  (1)  to  represent  a 
structured  design  process  that  integrates  all  related  pertinent  diagnostics  ele¬ 
ments,  (2)  to  represent  an  acquisition  approach  that  develops  and  acquires  var¬ 
ious  diagnostics  elements  as  a  package,  and  (3)  to  represent  a  deliverable  system 
(or  subsystem)  that  integrates  diagnostic  elements  [Brown  90a]. 


While  the  IDA  study  team  prefers  the  definition  put  forward  by  William  Keiner  [Keiner 
90],  it  felt  that  the  underlying  premise  behind  integrated  diagnostics  is  well  understood  even 
with  the  lack  of  a  clear  definition:  that  is,  to  improve  the  integration  of  diagnostics  elements 
that  are  treated  as  discrete  elements  to  be  developed  and  contracted  separately,  and  to  foster  fur¬ 
ther  centralized  management  of  independently  controlled  diagnostic  elements. 

Also  apparent  to  the  IDA  study  team  was  a  general  consensus  at  the  1996  workshop  that 
significant  benefits  were  achievable  by  adopting  approaches  to  integrate  diagnostic  element 
capabilities  and  responsibilities,  and  that  these  benefits  resulted  from  the  total  diagnostic  capa¬ 
bility  exceeding  the  upper  performance  limits  of  individual  diagnostic  elements  acting  in  iso¬ 
lation. 

In  the  opinion  of  the  IDA  study  team,  an  approach  to  solving  the  definitional  issue  is  an 
open  system  architectural  approach  that  would  link  together  the  general  concerns  and  defini¬ 
tional  differences  highlighted  in  the  1989  workshop  quotation.  Such  an  open  system  architec¬ 
ture  should  be  designed  with  features  that  facilitate  the  following; 

•  A  structured  design  process  to  integrate  all  relevant  diagnostics  elements. 

•  A  performance-based  acquisition  approach  for  the  delivery  of  diagnostics  elements 
as  a  package. 

•  Mechanisms  to  support  easy  integration  (e.g.,  plug-and-play  approaches)  of  sub¬ 
systems  that  will  improve  diagnostics  and  maintenance. 

If  such  an  architecture  were  established  and  followed,  many  of  the  definitional  concerns 
cited  at  this  workshop,  as  well  as  others  in  the  past,  would  be  resolved.  The  next  observation 
addresses  an  open  system  architectural  concept  that  would  meet  the  need  to  facilitate  multi-Ser- 
vice  and  logistic  support  interoperability  and  to  support  multi-user  applications  such  as  indus¬ 
try  and  DOD,  and  acquisition  and  field  users. 

Open  system  architecture  for  integrating  diagnostics  elements 

Observation  2.  Integrated  diagnostics  implementations  are  best  achieved  by 

reducing  the  use  of  diagnostic  element  interfaces  that  are  implementation 

unique  and  increasing  the  use  of  interface  standards  that  are  open,  well  defined, 

non-proprietary,  and  commonly  accepted. 

Current  legacy  system  support  and  mmntenance  infrastructures  will  remain  as  barriers 
to  effective  diagnostic  element  integration  and  reuse  until  these  infrastructures  move  away 
from  fixed  design  characteristics.  Improvements  will  come  in  the  move  towards  industry  stan- 


dards  body  or  non-proprietary  open  system  architectural  interfaces  and  de  facto  standards 
defining  performance  (functional  and  physical)  requirements  for  interconnecting  diagnostics 
elements. 

Effective  implementation  of  integrated  diagnostics  requires  the  integration  of  diagnos¬ 
tic  elements  across  weapon  system  and  support  infrastructures.  Currently,  the  capability  of 
existing  infrastructures  to  accommodate  improved  diagnostic  elements  is  poor  unless  the  ele¬ 
ments  are  uniquely  designed  for  the  specific  architectural  interface.  The  opportunities  for  reuse 
and  integration  of  existing  diagnostic  elements  that  were  designed  with  system-peculiar  inter¬ 
faces  are  equally  poor  unless  the  alternate  application  has  a  very  similar  or  identical  interface. 

Common  architectures  that  have  an  open  systems  approach  to  diagnostic  element  inter¬ 
faces  and  that  are  widely  applicable  across  weapon  systems  and  support  infrastructures  could 
provide  the  following  benefits: 

•  An  improvement  in  diagnostics  and  maintenance  support  performance. 

•  Greater  opportunity  for  DOD  to  be  one  of  many  customers  in  the  marketplace  for 
individual  diagnostic  element  solutions  and  open  the  opportunity  for  competitive 
solutions  from  multiple  contractors. 

•  A  reduction  in  costs  during  the  early  part  of  the  system’s  implementation  phase 
since  some  of  the  diagnostic  elements  are  likely  to  exist  and  should  be  easy  to  inte¬ 
grate. 

•  An  increase  in  system  design  and  support  flexibility. 

•  Easier  improvements  in  incremental  technology  and  systems. 

Turning  data  into  diagnostics  information 

Observation  3.  For  integrated  diagnostics  to  be  beneficial  and  effective,  col¬ 
lected  data  must  be  turned  into  information  that  is  accurate,  timely,  reliable,  and, 

most  importantly,  useful  in  helping  to  predict  and  eliminate  or  reduce  field  re¬ 
pair  requirements. 

The  need  for  data  to  be  accurate,  timely,  and  reliable  was  continuously  cited  during 
workshop  sessions.  Other  desirable  attributes  involving  integrated  diagnostics  data  cited  at  the 
workshop  included  easily  accessible  automated  capture  or  exchange,  and  minimal  cost  for  data 
capmre,  storage,  and  retrieval. 


Yet  surprising  by  its  omission  was  the  requirement  for  data  to  be  predictive  and  useficl. 
The  IDA  study  team  believes  this  was  an  oversight  and  that  workshop  participants  generally 
assumed  data  would  be  useful.  However,  the  IDA  study  team  observed  that  for  data  to  be  use¬ 
ful,  they  need  to  be  turned  into  information  that  supports  maintenance  and  systems  repair. 
Therefore,  useful  data  must  include  the  following  attributes: 

•  Accurately  portrays  a  number  of  status  conditions  (e.g.,  fault  detection,  fault  isola¬ 
tion,  fault  prediction,  system  configuration). 

•  Identifies  preferred  action  strategies  (e.g.,  corrective  procedures,  diagnostic 
approaches,  reliability  enhancement  options). 

•  Assesses  capabilities  and  identifies  deficiencies  (e.g.,  analyzes  high-cost  drivers, 
supports  “what  if’  analyses,  operating  performance  feedback,  spares  requirements). 

Integration  and  analysis  functions  that  turn  raw  data  into  useful  information  may  be 
separate  tools  or  capabilities  incorporated  within  diagnostic  elements.  In  either  case,  the  IDA 
study  team  recognized  potential  benefits  if  this  functional  capability  were  modularized  and 
implemented  as  part  of  an  open  system  architecture  for  integrated  diagnostics  elements. 

3.3  Recommendation 

The  IDA  study  team  set  out  to  identify  key  integrated  diagnostics  improvements  or 
activities  that  would  have  the  greatest  potential  of  broadly  enhancing  DOD  weapons  systems 
maintenance  and  support  capabilities.  Although  a  number  of  improvement  initiatives  were  pos¬ 
tulated,  the  establishment  of  an  open  system  architecture  for  integrated  diagnostics  singularly 
stood  out  as  the  IDA  study  team’s  recommendation  with  the  greatest  potential  benefits.  There¬ 
fore,  independent  of  the  workshop,  the  IDA  study  team  recommended  the  following: 

Recommendation.  DOD  should  conduct  a  two-phase  study  and  demonstration 

activity  to  establish  an  integrated  diagnostics  open  system  architecture. 

•  Phase  I.  Initiate  a  broad-based  study  effort  to  identify  and  characterize  key 
diagnostic  interfaces. 

•  Phase  n.  Develop  and  conduct  a  demonstration  of  an  open  system  architec¬ 
ture  approach  for  integrating  diagnostic  elements  across  weapon  systems 
and  support  infrastructures. 

Phase  I  activity  should  identify  key  diagnostic  interfaces,  both  in  current  use  and  for 
proposed  future  systems.  All  identified  diagnostic  interfaces  should  be  fiilly  characterized  in 


terms  of  functional  and  physical  attributes.  Interface  attributes  should  include  hardware  and 
software  details  that  address  form,  fit,  function,  protocols,  performance  boundaries,  and  a  sta¬ 
tus  of  the  interface  specifications  or  standards. 

Phase  n  activity  should  develop  an  open  system  architecture  concept  to  interconnect, 
exchange  data,  and  process  information  between  diagnostic  elements.  The  development  activ¬ 
ity  should  culminate  in  implementation  of  the  open  system  architecture.  The  architecture 
should  demonstrate  diagnostic  element  integration  and  cross-implementation  of  integrated 
diagnostics  functions  on  several  weapons  systems.  Weapon  systems  candidates  for  demonstra¬ 
tion  should  come  from  different  Services,  and  each  candidate  should  rely  on  different  infra¬ 
structures  for  support  and  maintenance.  Finally,  the  open  system  architecture  should  include 
features  discussed  previously  under  Observation  1. 

Although  this  recommendation  does  not  appear  directly  in  the  workshop  minutes,  the 
IDA  study  team  felt  that  synthesis  of  the  information  provided  has  led  directly  to  this  recom¬ 
mended  approach.  Many  of  the  solutions  offered  in  Appendix  A  dealt  with  one  or  more  aspect 
of  an  open  system  architecture  definition,  and  workshop  participants  tended  to  focus  on  the 
individual  aspects  rather  than  the  underlying  problem:  the  lack  of  a  well-defined  framework  for 
all  of  the  diagnostic  problems.  In  the  minutes  of  the  workshop  (Appendix  A),  many  of  the  piec¬ 
es  of  an  open  system  integrated  diagnostics  architecture  were  discussed  while  not  addressing 
the  overall  consequences  of  a  clear  framework. 

The  Legacy  Systems  Working  Group,  for  example,  cited  non-standard  interfaces,  poor 
configuration  management,  and  non-standard  data  collection,  transfer,  and  storage.  Each  of 
these  is  a  symptom  of  a  poor  framework,  which  leads  to  proliferation  of  equipment,  software, 
and  maintenance  processes  that  create  obstacles  for  the  training  of  maintenance  personnel  and 
prevent  feedback  of  information  at  all  levels.  An  open  system  architecture  will  address  all  of 
these  problems.  The  group  further  cited  a  need  for  standard  approaches  instead  of  dictated  solu¬ 
tions,  which  implies  “open”  aspects.  Finally,  the  group  requested  a  clearinghouse  for  diagnostic 
information.  While  there  are  benefits  in  all  of  their  solutions,  the  complexity  of  the  multiple 
approaches  needs  to  be  simplified  and  made  available  to  all. 

The  Cross-Cutting  Working  Group  cited  a  very  similar  set  of  problems  and  solutions. 
For  example,  a  request  was  made  to  estabhsh  a  govemment/industry  consortium  to  resolve  def¬ 
initions,  interfaces,  and  measures  of  effectiveness.  The  group  also  cited  no  common  process  or 
ill-defined  processes,  the  mixing  of  personnel  and  organizational  objectives  with  integrated 
diagnostics,  and  non-common,  inaccurate,  and  incomplete  data  availabihty.  There  were  several 
technology-related  problems  such  as  engine  test  cell  data  or  analog  built-in  test,  but  both 


groups  as  a  whole  indicated  that  technology  was  not  the  problem  while  political,  cultural,  and 
organizational  difficulties  were. 

The  team  felt  that  each  of  the  problenK  cited  by  the  workshop  participants  were  real, 
and  in  many  cases  the  solutions  could  offer  considerable  savings  on  their  own  merit.  However, 
the  synergy  that  an  open  system  architecture  approach  to  integrated  diagnostics  could  provide 
would  far  outweigh  the  individual  gains  of  the  specific  workshop  solutions. 


APPENDIX  A.  WORKSHOP  MINUTES 


[Note;  The  minutes  have  been  approved  by  the  chairs  of  the  two  working  groups  and  distrib¬ 
uted  to  workshop  participants  prior  to  the  publication  of  this  IDA  paper.  Therefore,  no  edito¬ 
rial  changes  have  been  made.] 


The  Integrated  Diagnostics  Workshop  was  conducted  by  and  hosted  at  the  Institute  for 
Defense  Analyses  (IDA)  on  8  August  1996.  BDA’s  participation  was  sponsored  by  the  Office 
of  the  Deputy  Under  Secretary  of  Defense  (Industrial  Affairs  and  Installations),  Industrial 
Capabilities  and  Assessments  (IC«&A)  Directorate.  Mr.  Herb  Brown,  from  IDA,  served  as 
workshop  chairperson.  The  workshop  agenda  and  introductory  charts  are  presented  in  Appen¬ 
dix  C. 


A.1  INTRODUCTION  &  WORKSHOP  OVERVIEW 

Mr.  Brown  began  the  workshop  by  welcoming  the  participants  (listed  in  Appendix  C); 
and  noting  they  represented  a  broad  cross-section  of  technology  development,  acquisition,  and 
support  functional  areas  from  the  four  Services  (Army,  Navy,  Air  Force,  and  Marine  Corps)  and 
OSD  (Office  of  the  Secretary  of  Defense).  He  provided  both  the  administrative  details  and  an 
overview  of  the  workshop. 

This  was  followed  with  an  introduction  to  the  workshop  by  Ms.  Christine  Fisher,  from 
IC&A.  She  provided  an  overview  of  immediate  workshop  objectives  and  outlined  a  vision  of 
potential  next  step/future  objectives: 

a.  Immediate  Objectives: 

1 .  Increase  awareness  of  benefits  of  integrated  diagnostics  applications. 

2.  Identify  support  problems  where  a  better  approach  can  be  applied. 

3.  Propose  cross-cutting  integrated  diagnostics  opportunities. 

b.  Next  Step: 

1.  Based  on  workshop,  define  pervasive  diagnostic  issues. 

2.  Explore  open/non-proprietary  architecture  opportunities. 

Ms.  Fisher  reviewed  the  history  and  progress  of  integrated  diagnostics  initiatives, 
emphasized  technology  opportunities,  and  discussed  both  the  opportunities  and  challenges  of 
the  new  acquisition  environment.  She  next  challenged  the  attendees  by  commissioning  the 
workshop  to  identify  perceived  problems  and  new  directions.  Copies  of  Ms.  Fisher’s  ID  Work¬ 
shop  Overview  are  presented  in  Appendix  D. 

A.2  BACKGROUND  &  CURRENT  ID  INmATTVES 

As  part  of  the  workshop’s  goal  to  raise  the  level  of  consciousness  and  thinking  about 
integrated  diagnostics,  Mr.  Brown  introduced  presentations  on  four  current  initiatives  directly 


involving  integrated  diagnostics  elements  and  capabilities.  This  was  not  an  attempt  to  be  a 
complete  review  of  initiatives;  instead,  it  was  intended  as  a  mechanism  for  permitting  rapid 
transition  to  serious  working  group  discussions  of  test  and  diagnostics  problems  and  potential 
solutions.  The  following  lists  the  initiatives/programs,  the  presenters,  and  the  appendix  loca¬ 
tion  of  the  respective  briefing  charts: 

a.  Integrated  Maintenance  Data  System  (IMDS),  by  Maj  Bryn  Turner,  Appendix  E. 

b.  Integrated  Diagnostics  for  JAST,  by  Mr.  Gary  Smith,  Appendix  F. 

c.  Aircraft  Maintenance  Environment,  by  Mr.  Martin  Bare,  Appendix  G. 

d.  Automatic  Test  Systems  (ATS)  R&D,  by  Mr.  Harry  McGuckin,  Appendix  H. 

Next  in  the  agenda,  there  had  been  scheduled  a  short  summation  of  test  and  diagnostic 
problems  and  potential  solutions  that  were  submitted  to  IDA  per  the  workshop  invitation  letter. 
Because  the  scheduled  activities  were  rurming  longer  than  plarmed  at  this  point  and  because 
copies  of  submitted  problems  and  solutions  had  already  been  sent  to  attendees  as  part  of  a 
“read-ahead”  information  package,  Mr.  Brown  elected  to  move  directly  into  the  working-group 
sessions. 

A.3  WORKING-GROUP  SESSIONS 

The  structure  of  the  working-group  sessions  was,  by  design,  very  flexible  —  the  intent 
was  to  stimulate  new  views  and  new  ideas.  To  further  foster  open-mindedness  and  creative 
thinking,  co-chairpersons  of  the  working-group  sessions  were  selected  from  each  of  the  four 
services.  The  attendees  were  asked  to  select  one  of  two  areas  of  concentration: 

a.  Working-Group  A:  Focus  on  Legacy  Systems  Problems/Solutions 

b.  Working-Group  B:  Address  Cross-cutting  Problems/Solutions  Between  Domains 

Initially,  each  of  the  working-groups  convened  to  discuss  perceived  problems.  After 
about  2-hours  of  discussions,  all  of  the  workshop  attendees  were  reconvened  and  results  of  this 
initial  session  were  shared  between  both  working-groups.  Next,  the  workshop  broke  into  the 
two  working-groups  again  to  address  potential  solutions.  Finally,  all  of  the  attendees  recon¬ 
vened  to  summarize  results  of  the  second  working-group  session,  and  to  open  discussions 
among  all  workshop  attendees.  The  following  summarizes  the  results  of  these  working-group 
activities  by  the  working-group  concentration  areas. 


A.3.1  FOCUS  ON  LEGACY  SYSTEMS  PROBLEMS/SOLUTIONS  (WORKING- 
GROUP  A) 


This  working-group  was  co-chaired  by  Mr.  Mike  Heilman  of  the  Marine  Corps  and  Mr. 
Pat  Stevens  of  the  Army.  Mr.  August  (Gus)  Scalia  of  IDA  served  as  the  recorder.  Other  partic¬ 
ipants  in  this  group  are  listed: 


SMSgt  Greg  Brewer 
Mr.  Jeff  Dean 
Mr.  Doug  DuBois 
TSgt  Greg  Greening 
Mr.  Tom  Ingram 
Mr.  Dave  Paros 
Mr.  Bruce  Scott 
Dr.  Li  Pi  Su 


Mr.  Vem  Chance 
Dr.  Warren  Debany 
Mr.  Charles  Gelfenstein 
LtCol  Harry  Hamilton 
Mr.  Mukund  Modi 
Mr.  John  Powell 
Mr.  Butch  Sneade 


A3.1.1  Problems  Session:  Legacy  Systems 

After  an  initial  round  of  introductions  by  members  of  the  legacy  systems  working- 
group,  the  participants  began  to  discuss  what  they  felt  were  the  overall  objectives  of  the  inte¬ 
grated  diagnostic  workshop.  It  was  agreed  this  would  help  them  in  deciding  their  approach  to 
identifying  problem  statements  and  how  they  could  best  be  tied  to  the  overall  objectives  of  this 
workshop.  Group  consensus  was  that  they  should  shoot  to  develop  a  top  10  type  list  that  could 
result  in  some  action  being  taken  by  OSD.  After  a  relatively  brief  discussion  period  it  was 
agreed  their  purpose  was  to  increase  appreciation  of  the  benefits  of  integrated  diagnostics 
applications,  to  identify  support  problems  where  a  better  approach  can  be  applied,  and  to  pro¬ 
pose  cross-cutting  integrated  diagnostics  solutions  to  resolve  the  problems. 


The  working-group  then  began  to  develop  a  list  of  both  design  data  and  test  diagnostics 
problems  focused  to  legacy  systems.  As  the  group  activity  began,  the  co-chairs  mentioned  that 
this  effort  should  be  done  within  some  boundaries  and  the  group  agreed  that  they  should  con¬ 
sider  all  existing  technology  opportunities,  new  acquisition  reform  initiatives  taken  with  the 
recent  release  of  the  new  DoD  5000  series  instructions  and  regulations,  and  the  work  recently 
done  on  defense  acquisition  deskbooks. 

The  first  issue  to  be  raised  in  the  process  of  identifying  problem  one  was  to  decide 
whether  the  term  legacy  system  meant  an  information  system  or  weapon  system.  It  was  pointed 
out  by  several  members  that  the  most  recent  OSD  integrated  diagnostic  funding  efforts  seemed 
to  be  focused  in  the  area  of  information  systems  and  databases  etc.  rather  than  weapon  plat- 
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forms.  The  group  discussed  some  of  the  ID  tasks  that  were  briefed  just  prior  to  this  breakout 
session,  then  agreed  that  both  areas  were  appropriate.  The  decision  was  to  made  test  the  prob¬ 
lem  statements  so  that  both  weapon  system  diagnostic  problems  and  the  transfer  abiUty  of  diag¬ 
nostic  information  was  identified.  The  co-chair,  Pat  Stevens,  then  took  control  of  the  group 
from  Mr.  Mike  Heilman,  and  began  the  session  by  asking  for  problem  ideas  the  group  felt  had 
resulted  in  not  achieving  either  real  dollar  savings,  lower  maintenance  manhours  per  operating 
hour,  or  the  desired  reliability  from  their  products  as  a  result  of  poorintegrated  diagnostics. 

As  problems  were  discussed,  one  issue  was  brought  up  that  did  not  make  the  top  10 
problems  list  (principally  because  it  did  not  directly  relate  to  legacy  systems).  However,  this 
issue  is  recorded  here  as  the  group  felt  it  was  important.  Issue  Statement:  The  policy  level  folks 
do  not  appear  to  be  included  sufficiently  in  the  support  process;  and  some  of  the  current  policies 
are  not  based  on  tested  or  evaluated  field  results,  but  appear  to  be  designed  to  resolve  global 
program  and  acquisition  management  issues. 

The  following  listing  presents  the  identified  problem  statements  in  italics  and  adds  any 
clarifying  or  additional  comments  provided  by  the  working-group  in  regular  font  type. 

Problem  1;  Platforms  are  non-standard  in  hardware  and  software  at  the  function  lev¬ 
els,  the  system  level,  platform  and  inter-Service.  It  was  noted  that  test  and  diagnostic 
interfaces  are  not  standardized  at  any  level. 

Problem  2:  There  is  a  lack  of  design  data  and  test  documentation.  There  is  also  poor 
configuration  management  for  the  procured  design  data  and  test  documentation  when 
related  to  equipment  changes  (e.g.,  hardware,  software,  firmware,  and  paper).  Diag¬ 
nostics  are  either  not  bought,  or  when  they  are  bought,  they  are  not  in  a  useable  format. 
Available  data  may  not  have  been  maintained  (there  is  a  lack  of  configuration  control 
between  equipment  and  data). 

Problem  3:  Data  collection,  data  distribution  or  databases  are  non-standard  for  oper¬ 
ational,  maintenance,  and  configuration  data  and  the  validity  of  the  data  is  often  ques¬ 
tionable. 

Problem  4:  The  number  of  maintainers  and  the  amount  of  formal  training  they  receive 
is  decreasing.  This  is  resulting  in  an  increased  need  for  field  on-the-job-training  ( OJT), 
generalized  training  and  experience  level  training. 


Problem  5:  Streamlining  acquisition  actions  are  not  yet  well-defined  nor  well-applied 
for  legacy  type  systems  in  the  areas  of  implementing  COTS  and  commercial  parts  prod¬ 
uct  into  old  systems.  There  is  no  demonstrated  process  for  linking  diagnostics  improve¬ 
ments  for  COTS  and  commercial  parts. 

Problem  6:  Maintenance  equipment  and  methods  are  not  being  tracked  with  actual 
system  failures.  The  feedback  loop  from  the  field  is  missing. 

Problem  7:  Operational  availability  is  being  documented  higher  than  that  which  is 
really  achieved  and  the  result  is  that  we  do  not  get  sufficient  support  dollars  budgeted 
for  field  support. 

Problem  8:  TTiere  is  a  lack  of  low  level  capabilities  (e.g.,  BIT fault  grading,  interfaces, 
etc.)  to  support  higher  level  maintenance  goals  (e.g.,  time  to  repair,  CND/RTOK,  etc.) 

Problem  9:  Some  general  solutions  do  not  fit  for  implementation  on  the  older  legacy 
type  systems  and  some  single  point,  single  platform  or  Service  solutions  are  indeed 
good  solutions  (solutions  do  not  always  have  to  be  general  in  nature  and  always  be 
applied  across  the  board  to  all  the  Services  to  be  the  correct  answer). 

Problem  10:  The  department  does  not  have  an  Integrated  Diagnostic  clearing  house 
(there  is  no  central  point-of-contact  (POC)  or  group  for  integrated  diagnostics  where 
you  can  go  to  obtain  the  latest  ID  information).  The  question  is  -  How  do  we  effectively 
transfer  ID  information? 

Problem  11:  There  is  an  immediate  need  for  common  commercial  equivalents  and/or 
replacements  of  necessary  defense  interface  specifications  and  standards  (e.g.  stan¬ 
dards  containing  processes  such  as  those  found  in  MIL-STD-I814)  for  interface  speci¬ 
fications,  infrastructure,  and  functionality.  They  should  also  be  referenced  for  guidance 
in  the  new  defense  acquisition  deskbooks  and  handbooks. 

At  this  point,  the  problem  identification  session  for  the  legacy  system  area  was  com¬ 
pleted  and  all  workshop  attendees  then  re-convened  in  the  main  conference  room  to  share  the 
results  of  the  breakout  session.  The  results  were  then  presented  by  Mr.  Pat  Stevens  for  working- 
group  A. 


A.3.1.2  Solution  Session:  Legacy  Systems 


After  the  joint  problem  session,  the  working-group  then  re-assembled  to  begin  the  sec¬ 
ond  part  of  the  workshop  where  they  would  recommend  problem  solutions.  Participants  began 
this  session  with  a  long  discussion  of  the  comments  and  questions  received  during  the  joint  ses¬ 
sion  problem  briefings. 

The  group  then  commenced  reading  the  problems  and  regrouping  them  based  on  their 
potential  solution.  The  following  listing  summarizes  how  the  statements  were  combined.  For 
traceability,  the  original  problem  numbers  are  presented. 

•  Problem  Statement  1  remained  the  same. 

•  Problem  Statement  2  remained  the  same. 

•  Problem  Statement  3  &  7  merged. 

•  Problem  Statement  4  remained  the  same. 

•  Problem  Statement  5  &  1 1  merged. 

•  Problem  Statement  6  &  8  merged. 

•  Problem  Statement  9  remained  the  same. 

•  Problem  Statement  10  remained  the  same. 

Problem  Statement  1:  Platforms  are  non-standard  in  hardware  and  software  at  the 
junction  levels,  the  system  level,  platform  and  inter-Service.  It  was  noted  that  test  and 
diagnostic  interfaces  are  not  standardized  at  any  level. 

No  solution  proposed. 

Problem  Statement  2:  There  is  a  lack  of  design  data  and  test  documentation.  There  is 
also  poor  configuration  management  for  the  procured  design  data  and  test  documenta¬ 
tion  when  related  to  equipment  changes  (e.g.,  hardware,  software,  firmware,  and 
paper).  Diagnostics  are  either  not  bought,  or  when  they  are  bought,  they  are  not  in  a 
useable  format.  Available  data  may  not  have  been  maintained  (there  is  a  lack  of  config¬ 
uration  control  between  equipment  and  data). 

No  solution  proposed. 

Problem  Statement  3:  Data  collection,  data  distribution  or  databases  are  non-stan¬ 
dard  for  operational,  maintenance,  and  configuration  data  and  the  validity  of  the  data 


is  often  questionable.  (Problem  Statement  7;)  Operational  availability  is  being  docu¬ 
mented  higher  than  that  which  is  really  achieved  and  the  result  is  that  we  do  not  get 
sufficient  support  dollars  budgeted  for  field  support. 

No  solution  proposed. 

Problem  Statement  4:  The  number  of  maintainers  and  the  amount  of  formal  training 
they  receive  is  decreasing.  This  is  resulting  in  an  increased  need  for  field  on-the-job- 
training  ( OJT),  generalized  training  and  experience  level  training. 

Recommended  Solution:  New  technology  demonstrations  should  be  conducted  to 
show  where  technology  can  make  up  for  formal  classroom  training  deficiencies.  These 
should  be  based  upon  the  collected  results  of  current  and  past  demonstrations. 

Problem  Statement  5:  Streamlining  acquisition  actions  are  not  yet  well-defined  nor 
well-applied  for  legacy  type  systems  in  the  areas  of  implementing  COTS  and  commer¬ 
cial  parts  product  into  old  systems.  There  is  no  demonstrated  process  for  linking  diag¬ 
nostics  improvements  for  COTS  and  commercial  parts.  (Problem  Statement  11:)  There 
is  an  immediate  need  for  common  commercial  equivalents  and/or  replacements  of  nec¬ 
essary  defense  interface  specifications  and  standards  (e.g.  standards  containing  pro¬ 
cesses  such  as  those  found  in  MILrSTD-1814)  for  interface  specifications, 
infrastructure,  and  functionality.  They  should  also  be  referenced  for  guidance  in  the 
new  defense  acquisition  deskbooks  and  handbooks. 

Recommended  Solution:  A  list  of  recorrunended  standards  should  be  compiled,  and 
they  should  be  converted  to  commercial  form  and/or  included  into  the  new  DoD  or  Ser¬ 
vice  acquisition  deskbooks. 

Problem  Statement  6:  Maintenance  equipment  and  methods  are  not  being  tracked 
with  actual  system  failures.  The  feedback  loop  from  the  field  is  missing.  (Problem 
Statement  8:)  There  is  a  lack  of  low  level  capabilities  (e.g.,  BIT  fault  grading,  interfac¬ 
es,  etc.)  to  support  higher  level  maintenance  goals  (e.g.,  time  to  repair,  CND/RTOK, 
etc.) 

Recommended  Solution:  Improve  data  collection  and  tracking  and  utilize  to  imple¬ 
ment  diagnostic  improvements. 

Problem  Statement  9:  Some  general  solutions  do  not  fit  for  implementation  on  the  old¬ 
er  legacy  type  systems  and  some  single  point,  single  platform  or  Service  solutions  are 


indeed  good  solutions  (solutions  do  not  always  have  to  be  general  in  nature  and  always 
be  applied  across  the  board  to  all  the  Services  to  be  the  correct  answer). 

Recommended  Solution:  Organization  and  facility  established  to  bring  Integrated 
Diagnostic  problems/solutions  to  (e.g.  to  trade-off  general  verses  point  solutions  etc.) 

Problem  Statement  10:  The  department  does  not  have  an  Integrated  Diagnostic  clear¬ 
ing  house  (there  is  no  central  point-of-contact  (POC)  or  group  for  integrated  diagnostics 
where  you  can  go  to  obtain  the  latest  ID  information).  The  question  is  -  How  do  we 
effectively  transfer  ID  information  ? 

Recommended  Solution:  Develop  a  draft  strategic  plan  for  Integrated  Diagnostics. 
Start  a  tri-Service  group  on  an  ID  level  (e.g.,  IPT,  IRB,  Management  Board  etc.)  similar 
to  actions  recently  taken  by  DoD  for  Automatic  Test  Systems  (ATS)  management.  The 
department  needs  to  find  a  method  to  move  more  pieces  of  what  each  Service  is  doing 
right  and  communicate  that  immediately  to  the  other  Services. 


Late  into  the  second  work-group  session  Mr.  Brown  came  in  and  stated  that  the  first 
group  was  complete  and  asked  that  we  rejoin  the  other  workshop  members  for  the  summary 
and  wrap-up  session.  Again,  Mr.  Pat  Stevens  from  the  Army  presented  the  results  of  working- 
group  A  to  the  entire  group. 


A.3.2  ADDRESS  CROSS-CUTTING  PROBLEMS/SOLUTIONS  BETWEEN 
DOMAINS  (WORKING-GROUP  B) 


This  working-group  was  co-chaired  by  Mr.  Martin  Bare  of  the  Navy  and  Mr.  Gary 
Smith  of  the  Air  Force.  Dr.  William  R.  Simpson  of  IDA  served  as  the  recorder.  Other  partici¬ 
pants  in  this  group  are  listed. 


Mr.  Timothy  Bearse 
Mr.  Herb  Brown 
Mr.  Stephen  Hull 
Mr.  David  Kidd 
Mr.  Harry  McGuckin 
Mr.  Jeff  Riggs 
Mr.  Howard  Savage 
Mr.  John  Schroeder 
Maj  Bryn  Turner 


Mr.  Charles  Bosco 
Mr.  Bill  Horth 
Mr.  Bob  Johnson 
Mr.  Terry  Lindemarm 
MSgt  Joe  Oram 
Mr.  Bill  Ross 
Mr.  Rickey  Schippang 
Mr.  Alex  Smimow 
Capt  Gary  Wiley 


A.3.2.1  Problems  Session:  Cross-Cutting  Domains 

After  a  brief  introduction  of  working-group  participants,  the  entire  group  entered  into 
lively  discussions  of  issues.  This  first  session  was  to  provide  statements  of  problems  involving 
integrated  diagnostics.  This  working-group  considered  those  issues  that  were  perceived  to  rep¬ 
resent  cross-cutting  problems  between  and  among  domains  (that  is,  spaimed  legacy  to  new 
designs  and  inter-service  applications  such  as  air,  land,  and  sea). 

This  first  working-group  session  was  an  open  forum  with  no  restrictions  and  not  for 
attribution.  It  turned  into  a  brainstorming  session;  and  as  a  result,  the  many  highlighted  prob¬ 
lems  do  not  follow  any  specific  flow.  The  following  listing  presents  the  identified  problems  in 
italics,  and  follows  in  regular  font  with  the  essence  of  extended  discussions  when  appropriate: 

Problem  1:  No  central  point  of  focus  for  id,  duplicate  expenditures  because  of  this  lack 
of  communication.  It  was  noted  that  we  have  little  conununication  between  services.  In 
fact  a  central  focus  point  might  help. 

Problem  2:  Technical  transition  is  not  being  handled  well.  It  was  noted  that  some  id 
related  things  are  flying  on  the  111  but  not  yet  available  to  the  services. 

Problem  3:  R&D  not  sent  out  into  the  field.  Detractors  include  Planning  Programming 
and  Budgeting  System  (PPBS),  hand  off  from  R&D  to  program  office,,  rice  bowls,  etc. 

Problem  4:  Coordinated  approaches  are  stymied  by:  Rice  Bowls,  Funding  lines,  and 
Administrative  Burdens.  There  is  no  referee  for  problems  among  services.  Often  can’t 
place  a  desirable  technology  in  the  right  funding  slots. 

Problem  5:  Priorities  are  not  the  same  among  the  services'  weapons  systems  pro¬ 
grams. 

Problem  6:  Rigidity  of  funding  deadlines,  sometimes  on  a  short  fuse  prevents  flexible 
use  of  funds  for  id  improvements. 

Problem  7:  Inability  to  control  funding  (i.e..  Congress)  makes  providing  Id  improve¬ 
ments  difficult. 

The  group  then  began  to  look  to  the  future  battle  force.  The  following  observations  are 
pertinent  and  shape  the  further  discussion  of  id:  All  battles  are  joint.  Communications  not  cur¬ 
rently  compatible,  and  Joint  logistics  is  needed  but  not  currently  supported. 


Problem  8:  Need  joint  logistics  approach:  interchangeable  parts,  intertwined  logistics 
lines,  redirect  logistics  in  transit,  just  in  time  logistics  approaches,  reliability  centered 
maintenance,  etc.  It  was  noted  that  the  idea  of  autonomous  logistics  as  presented  by  the 
JAST  review  is  attractive,  but  has  had  some  difficulty  in  software  assigning  peoples 
work  load  in  the  past. 

Problem  9:  Need  single  data  base  for  all  users. 

Problem  10:  Man-in-the-loop  is  slow  inaccurate.  A  future  approach  will  require  man 
out-of-the-loop  to  the  greatest  extent  possible.  Requirement  for  automation  here. 

Problem  11:  High  complexity  is  a  driving  factor  which  forces  man  out-of-the-loop. 

Problem  12:  Fault  tolerance  capabilities  add  to  systems  complexity.  Fault  tolerance 
creates  maintenance  problems  while  providing  performance  and  capability  to  weapon 
systems. 

Problem  13:  Inadequate  data  hiding  (possibly  not  providing  everything  to  everybody). 
Conflicts  somewhat  with  9,  and  is  characterized  by  information  glut  (providing  more 
information  than  can  be  used). 

Problem  14:  Each  program  has  a  different  urgency/latency  which  increases  complex¬ 
ity. 

Problem  15;  ID  not  understood.  Education  is  a  problem.  What  is  ID? 

Problem  16:  ID  is  after-the-fact.  Concurrent  engineering  is  what  is  needed.  System 
engineering  should  account  for  these  factors.  Problem  was  also  stated  as:  Requirements 
are  not  properly  or  thoroughly  stated. 

Problem  17:  Need  an  accepted  definition  of  ID. 

Problem  18:  We  often  forget  that  the  guy  in  the  cockpit  or  on  the  flight  line  have  the 
problems. 

Problem  19:  Systems  are  built  around  history  and  funding  rather  than  requirements. 
There  were  open  issues  on  how  this  will  influence  capabilities  given  what  we  expect  to 
see  in  the  future:  wider  range  of  platforms  with  some  commonality,  maintainor  versed 


in  a  wide  range  of  weapon  systems,  combined  MOSs  (already  begun  in  Army),  and  core 
competencies  too  broad. 

Problem  20:  Look  and  feel  of  maintenance  subsystems  not  same. 

Problem  21:  Functional  Requirements  that  cross  services  are  not  well  defined. 

•  Problem  22:  Need  tools  to  verify  that  diagnostics  are  met. 

Problem  23:  Contracting  not  flexible  with  penalties  and  incentives. 

Problem  24:  No  common  links  between/among  services.  Different  processing  results 
®  are  tied  to  process 


Problem  25:  Non-commonality  in  terms.  Need  common  set  of  definitions. 

^  Problem  26:  Cannot  articulate  our  processes  (acquisition,  development,  id,  etc.) 

A  discussion  of  the  virtual  test  bench  tool  followed  but  led  to  no  problem  statement. 

Problem  27:  Lack  of  methods  for  applying  ID. 

®  Problem  28:  Contract  specifications  and  measurements  are  not  satisfactory.  Not  all 

measures  of  effectiveness  (MOEs)  are  measurable.  Most  are  measurable  after  years  in 
the  field.  We  need  a  better  handle  on  the  quantitative  aspects  of  ID. 

#  Problem  29:  COTS  diagnostic  issues:  COTS  is  “trust  me  ”,  No  data  provided  with 

COTS,  NO  policing  of  claims,  and  Claims  are  exaggerated  in  commercial. 

Problem  30:  Small  lot  size  makes  items  expensive  and  hard  to  buy  commercially. 

®  Problem  31:  Long  development  time,  funding  problems,  etc.  leads  to  parts  obsoles¬ 

cence  often  before  the  weapon  system  is  fielded. 


Problem  32:  COTS  is  non-stationary  (rapidly  evolving  and  changing),  ID  needs  stabil¬ 
ity. 

Problem  33:  Kingdoms  and  Fiefdoms  in  development  and  sales.  PMAs,  SPOs,  take  on 
survival  lives  of  their  own  and  drive  problem  solutions. 

Problem  34:  State-of-the-art  is  moving  too  rapidly. 


A- 13 


Problem  35:  A  large  number  of  contract  issues  limit  our  ability  to  do  anything  integrat¬ 
ed. 

Problem  36:  Too  much  test  equipment  to  drag  around. 

Problem  37:  Verticality  between  levels  of  maintenance  not  well  defined. 

Problem  38:  Compartmentalized  problem  solutions  lead  to  integration  at  the  wrong 
levels. 

At  this  point  the  working-group  had  a  general  discussion  on  performance  based  specifications. 
The  following  concerns  were  observed:  F^I  can’t  push  down  requirements,  it  tends  to  give  up 
ownership  of  lower  levels,  a  lack  of  trust  frequently  exists,  visibility  into  the  system  is  often 
limited. 

Problem  39:  Giving  up  ownership  in  any  of  the  levels  is  hard. 

Problem  40:  Not  certain  where  level  of  ownership  in  data  is  needed. 

Problem  41:  No  performance  based  requirements  for  diagnostics.  Some  discussion  of 
where  standards  and  metrics  interact. 

The  co-chairpersons  cut  off  the  general  discussion  at  this  point.  They  then  asked  the 
working-group  to  categorize  these  problem  areas  so  that  they  might  be  synthesize  these  into 
problems  that  will  be  targets  for  solution.  The  chairs  developed  a  straw-man  list  of  topics  that 
evolved  into  the  list  below.  The  group  was  then  asked  to  place  each  of  these  problems  (by  num¬ 
ber)  in  categories.  Numbers  were  permitted  to  be  in  multiple  categories.  This  was  based  on  the 
perception  any  overlap  might  help  provide  emphasis  or  priority  to  the  discussions  and  help  to 
find  patterns  of  groups.  This  resulted  in  the  following  table. 


Table  A-1.  Synthesized  Listing  of  Perceived  Problems  by  General  Categories 


General  Problem  Categories 

Problem  Numbers  that  Apply 

1.  Contract  Issues 

3,  4,  5,  6,  7,  8,  13,  15,  16,  19,  21,  23, 
24,28,  29,  31,  32,  33,  35,  38,  39,  40,  41 

2.  Common  Definitions  and 

Standards 

4,  9,  15,  16,  17,  18,  19,  20,  21,  22,  25,  26, 

27,  28,  36,  37,  39,  41 

3.  Technical  Transition 

2,  3,  31,  34,  36 
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Table  A-1.  Synthesized  Listing  of  Perceived  Problems  by  General  Categories  (Continued) 


General  Problem  Categories 

Problem  Numbers  that  Apply 

4.  Process  and  Priorities 

4,  5,  6,  1,  13,  15,  16,  18,  19,  22,  23,  24, 

26,  27,  31,  32,  36,  37,  39,  40 

5.  Joint  Fighting  Force 

3,  4,  5,  8,  18,  20,  24,  31,  39,  40 

6.  Program  Differences 

1,  5,  14,  20,  24,  30,  41 

7.  Automation 

9.  10,  11,  12,  13,  20,  22,  34,  36,  38,  39,  40, 

41 

8.  Complexity 

9,  10,  11,  12,  14,  16,  19,  20,  31,  36,  37,  39, 

40 

9.  Contractor  Relations 

3,  4,  5,  19,  23,  29,  31,  33 

10.  COTS 

16,  19,  29,  31,  32,  34,  37 

11.  Training 

9,  10,  11,  13,  15,  20,  36,  37,  41 

12.  Look  and  Feel 

20 

13.  Stability  versus  Flexibility 

6,  7,  31,  32,  34,  39,  40 

14.  Compartmentalization 

15.  Ownership  Levels 

24,  29,  31,  39,  40 

16.  Organization 

1,  3,  4,  6,  7,  8,  10,  11,  17,  21,  24,  31, 

33,  37 

Based  on  the  results  presented  in  Table  4,  the  largest  number  of  problem  statements  fall 
in  the  following  six  categories: 

•  Contact  issues 

•  Common  Definitions  and  Standards 

•  Process  and  Priorities 

•  Organization 

•  Automation 

•  Complexity 

At  this  point  in  time,  the  problems  session  was  concluded,  and  all  workshop  attendees 
reconvened  to  share  the  results  of  the  two  working-groups. 
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A.3.2.2  Solution  Session:  Cross-Cutting  Domains 

This  working-group  then  set  about  the  task  of  identifying  solutions  to  the  problem  state¬ 
ments.  The  co-chairpersons  decided  that  the  working-group  should  try  to  synthesize  the  41 
problem  statements  developed  in  the  first  session  into  a  few  basic  combined  problem  state¬ 
ments.  This  was  discussed  at  length.  The  working-group  collectively  decided  to  start  with  the 
categories  of  highest  occurrence  and  synthesize  problems  and  then  propose  solutions.  This 
became  a  tangled  web  with  the  contract  issues,  common  definitions  and  process/priorities  all 
being  intertwined  at  almost  every  turn.  The  difficulty  appeared  to  be  the  lack  of  common 
frames  of  reference.  This  realization  then  led  to  the  following  synthesized  problem  statement 
below. 

Problem  1:  There  is  no  concurrence  on  common  definitions,  applicable  standards,  and 
usable  MOEsfor  ID. 

Although  there  was  agreement  regarding  this  problem,  almost  everyone  thought  this  was  too 
broad  to  tackle.  Therefore,  the  working-group  broke  this  problem  into  smaller  chunks. 

Problem  la:  There  is  no  concurrence  on  common  definitions. 

Problem  lb:  There  is  no  existing  agreed  to  sets  ofMOEs  for  ID  performance. 

Problem  Ic:  There  is  a  lack  of  commonly  agreed  to  functional  architecture  and  inter¬ 
faces  relative  to  ID. 

The  group  agreed  to  these  and  settled  on  a  solution  given  below. 

Solution  1:  Establish  (embellish  or  support)  a  govemment/industry  consortium  to 
resolve  these  three  problems.  Further,  the  consortium  must  be  joint,  involve  the  logistics 
and  design  communities,  and  probably  needs  to  be  a  funded  entity. 

Having  successfully  bridged  this  issue  the  group  tackled  processes.  It  was  noted  that 
there  was  a  lack  of  common  processes  for  ID.  The  working-group  observed  MIL-STD-1814 
and  MIL-STD-2165  provided  some  processes,  but  the  MIL-STDs  are  not  to  be  used  and  com¬ 
mercial  standard  do  not  address  these  issues.  This  led  to  the  following  problem  statement. 

Problem  2:  Lack  of  common  processes  to  institute  ID  ( Cradle  to  grave). 

A  vigorous  discussion  followed,  including  suggestions  to  prepare  drafts  or  to  partici¬ 
pate  in  standards  groups  such  as  IEEE  or  ANSI,  but  no  solution  was  put  forth.  A  suggestion 
was  made  to  examine  more  technical  issues  and  after  some  discussion,  the  following  example 
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was  given:  Radar  Cross  Section  verification  cannot  currently  be  done  in  less  than  a  hangar  size 
facility.  It  was  soon  realized  that  this  required  an  R&D  program  in  measurement  science  and 
was  out  of  scope  so  that  no  solution  was  proposed.  Similarly,  a  problem  statement  was  devel¬ 
oped  for  field  data.  The  following  was  agreed  to  by  the  working  group: 

Problem  3:  Field  data  can  be  reported  inaccurately  because  of  non-maintenance  use 

of  the  data,  (notably  the  reference  here  was  to  technician  fitness  reports) 

This  triggered  a  great  deal  of  discussion  with  general  agreement  that  this  could  be 
solved  with  forms  of  automated  data  collection,  and  the  following  was  given  as  a  solution. 

Solution  3:  Remove  subjectivity  in  field  maintenance  data  by  increasing  automated 

reporting. 

This  led  to  a  new  problem  statement  concerning  field  data  as  follows: 

Problem  4;  Current  data  collection  schemes  do  not  support  the  ID  process. 

An  example  was  provided  where  several  different  jet  engines  used  the  same  gas  path 
data  for  test  cell  trims,  but  each  had  a  different  data  unit  with  a  different  interface.  There  was 
some  discussion  that  the  general  problem  statement  could  be  expanded  to  cover  support  equip¬ 
ment  commonality  with  the  following  as  a  general  solution:  A  number  of  items  could  benefit 
from  support  equipment  commonality  or  standards  in  handling  aspects  of  support  equipment. 
After  much  discussion,  this  was  considered  out  of  scope  for  a  solution  statement.  A  great  deal 
of  energy  was  spent  here  in  discussion  but  it  was  finally  realized  that  the  details  of  the  specific 
application  were  too  important  to  propose  a  general  solution. 

Next,  the  working-group  noted  the  lack  of  organizations  and  processes  specifically 
chartered  with  addressing  ID  problems.  The  problem  statement  was  derived  as  follows: 

Problem  5;  There  is  no  institutionalization  of  the  ID  process. 

A  variety  of  solutions  were  considered:  the  establishment  of  joint  offices,  more  robust 
DoD  policy,  standardization  issues,  etc.  It  was  finally  agreed  that  ID  should  be  a  technical  dis¬ 
cipline  not  a  government  office  or  an  fmplementation.  The  working-group  then  noted  that  the 
ID  process  itself  is  not  well  defined.  Although  a  specific  solution  was  not  developed,  there  was 
general  concurrence  that  a  solution  should  adopt  joint  govemment/industry  approaches. 
Remarkably,  the  following  statement  was  made  during  the  working-group  discussions:  “ID  is 
not  a  technical  problem  —  it  is  a  political,  cultural,  organizational  problem”.  This  received 
some  support  and  no  refutation  here,  but  was  later  taken  issue  with  during  the  wrap-up. 


This  fostered  discussions  on  the  adequacy  of  built-in-test  (BIT)  capabiUties.  There  was 
general  agreement  that  digital  BIT  was  a  relatively  mature  technology.  However,  analog  BIT 
implementations  lag  behind  digital.  The  working-group  agree  that  most  of  the  difficulties  cen¬ 
tered  around  the  lack  of  common,  agreed-to,  methods.  Finally,  the  following  problem  statement 
was  developed: 

Problem  6;  Analog  BIT  lacks  protocols,  standards,  and  techniques  (methods). 

The  working-group  noted  that  an  approach  to  solving  this  problem  was  covered  in  an 
earlier  presentation  on  the  ATSR&D  program  (as  one  of  the  unfunded  tasks).  Therefore,  the  fol¬ 
lowing  solution  was  proposed: 

Solution  6:  Fund  ATS  R&D  Program  Task  #/i  (See  Mr.  McGuckin’s  presentation.) 

Having  used  the  allotted  time,  the  chairs  summarized  and  then  dismissed  the  working 
group  so  that  they  could  rejoin  the  workshop  for  summary  and  wrap-up. 


APPENDIX  B. 

WORKSHOP  AGENDA  AND  INTRODUCTORY  CHARTS 
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Group  A:  Focus  on  legacy  systems  problems/solutions 

Group  B:  Cross-cutting  problems/soiutions  between  domains 
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Propose  implementation  strategy 
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INTEGRATED  MAINTENANCE  DATA  SYSTEMS 


Major  Bryn  Dimer,  USAF 


Integrated  Maintenance  Data  System 


Integrated  Diagnostics  Workshop 
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Overview 
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Program  Description 
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Leverages  Commercial  Off  the  Shelf  (COTS)  software 
and  commercial  practices 


Target  Architecture 

(Global  view) 


Increments  of  Capability 


Total  Asset  Visibility 


System  Requirements 
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Increment  One 


Automated  and  manual  data  collection 
Interface  with:  lETM,  Diagnostics,  and  Equipment 


Increment  Two 
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What  is  the  plan? 

Notional  Increment  Three 


00 

0 

0 

© 

TJ 

•  pi^ 
© 

fl 

© 

©H 

C/5 

(XI 

Q 

4P 

es 

U 

CJ 

s 

*s 

© 

© 

ce 

pSh 

pQ 

(X 

© 

© 

*PM( 

© 

bX) 

0 

'o 

© 

fl 

ss 

pd 

•  mt 

© 

© 

el 

c/5 

hH  , 

1 

0 

g' 

© 

a  ^ 

«  Oi  £3 

i-j 

M  a  ^ 

«  ^  a 
.a  5^ 

c 

Lsi]  .S  C/) 

>20 

•S  ^  bC 


^  g 

3  § 

o  pfi 

Ph  Ifl 


>20 

•S  bD 

pSA  ^  > 

s  ^  ^ 

&-S  CJ 

i  s  "w 

•pm* 

^  ^  2 
c«  9 

a  H  §,!» 

fl  ^  2  S 
o  cc  ^  5 

•|SS:S 

§8=1 

€  w  CQ  *5 


fl 

li 

©  ^ 
^  fl 
©  o 
©  -C 
fl  © 
TS  fl 

a  «3 


S 

g  .s 

O  fl 
c«  *2 
b  A 
V  Ix 

Ph  H 


Interface  to  Command  and  Control  Systems 

Extend  Automated  WorkFlow  Manager  for 
maintenance  process 


Notional  Increment  Four 


o 


§ 

CQ 

fl 

o 

•2 

p 

S3 

fl 

O 

ts 

u 

Pn 

bl) 

P 

CAi 

HH 

P 

•  pp 

p 

P 

PM 

a 

O 


u 

^  i 

fl  .2 

^  2 
"2  S 

•r*  ” 
o  ^ 

Pm 


•s 

s  * 
2  « 
^  I 

"§  P 

«J  T3 

a>  sa 
S 

2  ® 

E 

«  'J-* 

a  ^ 

»  fl 

Pn  ^ 

e  I 

a>  .ST 
s 

(/) 


d  ^ 

^  ‘K 
d  ^ 

S  ^ 

Qi  ^ 
&£  ^ 

2  d 
d  0^ 

^  s 

?  M 
B  R 

•B  g 
2  § 

Sd  ^ 
ts  a 

d  ^ 

®  5 

u  a 


(ATLIS),  Stratotanker  Condition  Analysis  Logistics 
Evaluation  (SCALE),  Quality  Assurance  Tracking 
and  Tracking  (QANTTAS),  and  Material  Deficiency 
Reports  Expert  Analysis  System  (MDREAS) 


Notional  Increment  Five 


egin  ‘Wholesale”  Logistics  Integration 


What  is  the  plan? 

Notional  Increment  Five  (cont’d) 


Shut  Down  the  final  CAMS  system  (G054) 


Notional  Increment  Six 


Slide  13 


Contract  Information 


on 

U 

Pm  ^ 
hJ 

e£  > 
e  ® 

2  'S 

s  ® 

CA  CU 

fl  On 
o  a 

QJ  5« 

a  ^ 
'S 

<  © 

i;  « 

O  gj 

2  Q 


^  fS 
©  *> 
©  ^ 


© 

Q  ^ 

©  ^ 
5g  <13 

*©  K 


Q  Q 
h-3 

O  O 
O  O 


£  ^ 

«  s 

^  u 

<s  V, 


K  <s 
O  ® 

««  S 

0  ^ 

^  ■« 


>• 

©3 

Q  M 


a  2 

'S  9 

C/^  U 


Works  on  wide  mix  of  hardware 

Robust  training  (formal,  on-line  &  embedded) 


Program  Schedule 


00 

00 

ON 

ON 

ON 

ON 

VO 

00 

00 

ov 

ov 

ON 

ON 

U 

;n 

ov 

a\ 

ON 

ON 

tH 

vH 

pfi 

p& 

o 

#  \ 

o 
0  \ 

0 

0 

0 

P 

•  • 

•  • 

CO 

oe 

0^ 

CO 

ce 

A 

pfi 

pfi 

CO 

0^ 

o 

O 

'^Pii 

o 

> 

U 

0^ 

I  §  a 

I  a  2 

^  C3 

C3  «j  ® 

O  fa  fli 

U  1/2 


^  § 

05  *31 

»  .ss 

H  ^ 

■S  Q 

I  s 

I I 

s  ^ 

s 

o  ^ 
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An  automatic  test  system  (ATS)  comprises  three  components:  automatic 
test  equipment  (ATE),  the  stimulus  and  measurement  instruments,  oper¬ 
ating  system  software,  computer,  power  supplies,  and  interfaces;  test  pro¬ 
gram  sets  (TPSs),  the  interface  devices,  adapters,  software  programs  and 
documentation  to  test  individual  weapon  system  electronic  items  at  the 
box  and  circuit  card  level;  and  ATE/TPS  software  development  environ¬ 
ments,  the  ATE  and  unit  under  test  (UUT)  simulators  and  description  lan¬ 
guages,  and  programming  tools. 

The  stimulus  and  measurement  instruments,  operating  system  software, 
computer,  power  supplies,  and  interfaces. 

There  are  two  basic  types  of  weapon  system  built-in  test  (BIT).  Logic  lev¬ 
el  based  BIT,  also  identified  as  digital  BIT,  takes  advantage  of  digital 
pathways  to  concurrently  or  off-line  verify  performance  and  fault  isolate 
to  a  removable  item.  The  second  type  of  BIT  is  sensor  and  microcomputer 
based  and  can  be  applied  across  all  electronic,  electro-optical,  and  electro¬ 
mechanical  applications  [Rolfe  94,  p.  186]. 

Data  is  useful  when  it  (1)  accurately  portrays  a  number  of  status  condi¬ 
tions  (e.g.,  fault  detection,  fault  isolation,  fault  prediction,  system  config¬ 
uration);  (2)  identifies  preferred  action  strategies  (e.g.,  corrective 
procedures,  diagnostic  approaches,  reliability  enhancement  options);  and 
(3)  assesses  capabilities  and  identifies  deficiencies  (e.g.,  analyzes  high- 
cost  drivers,  supports  “what  if’  analyses,  operating  performance  feed¬ 
back,  spares  requirements). 

In  the  context  of  the  workshop,  the  term  demonstration  program  was 
intended  to  represent  an  activity  or  project  with  the  principal  goal  of  illus¬ 
trating  the  feasibility  and  benefits  of  integrating  diagnostics  elements 
[Rolfe  94,  p.  186]. 
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Diagnostics  is  the  practice  of  investigating  the  cause  or  nature  of  specific 
problems  that  inhibit  normal  operations.  System  diagnostic  capability  is 
developed  and  provided  through  engineering  design,  testing,  technical 
information,  and  trained  personnel.  The  diagnostics  elements  that  support 
this  capability  include  built-in  test,  automatic  and  manual  test  equipment, 
training,  maintenance  aiding,  and  technical  information  [Rolfe  94,  p. 
186]. 

In  the  context  of  the  workshop,  the  term  domain  was  intended  to  address 
weapon  systems  and  their  capabilities  that  span  from  legacy  systems  to 
new  and  proposed  weapon  system  designs,  and  included  all  inter-Service 
weapon  system  applications  across  operational  boundaries  such  as  those 
found  in  air,  land,  and  sea  system  applications. 

In  the  context  of  integrated  diagnostics,  the  IDA  study  team  defined  infra¬ 
structure  to  represent  a  collection  of  hardware  and  software  elements, 
interfaces,  policies,  and  processes  that  provide  the  means  of  implement¬ 
ing  a  support  capability. 

“. . .  a  structured  process  which  maximizes  the  effectiveness  of  diagnos¬ 
tics  by  integrating  the  individual  diagnostic  elements  of  testability,  auto¬ 
matic  testing,  manual  testing,  training,  maintenance  aiding,  and  technical 
information.”  [Keiner  90] 

“. . .  the  term  “Integrated  Diagnostics”  was  used  (1)  to  represent  a  struc¬ 
tured  design  process  that  integrates  all  related  pertinent  diagnostics  ele¬ 
ments,  (2)  to  represent  an  acquisition  approach  that  develops  and  acquires 
various  diagnostics  elements  as  a  package,  and  (3)  to  represent  a  deliver¬ 
able  system  (or  subsystem)  that  integrates  diagnostic  elements”  [Brown 
90a]. 

The  workshop  context  was  existing  systems. 

“All  phases  of  the  system’s  life  including  research,  development,  test  and 
evaluation,  production,  deployment  (inventory),  operations  and  support 
and  disposal.”  [DOD  91,  B-58] 

The  chain  of  logistics  support  that  goes  from  the  battlefield  back  to  the 
continental  United  States  maintenance  depots,  factories,  and  contractor 
support  personnel. 
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Provides  a  structured  design  process  to  integrate  all  relevant  diagnostics 
elements,  a  performance-based  acquisition  approach  for  the  delivery  of 
diagnostics  elements  as  a  package,  and  mechanisms  to  facilitate  easy  inte¬ 
gration  (e.g.,  plug-and-play  approaches)  of  subsystems  that  will  improve 
diagnostics  and  maintenance.  It  also  uses  well-defined,  industry  standards 
body,  or  commonly  accepted  interfaces  to  interconnect,  exchange  data, 
and  process  information  between  diagnostic  elements. 

A  condition  that  limited  weapon  system  capability  or  performance.  For 
the  purpose  of  this  analysis,  a  problem  was  considered  critical  when  it  was 
more  costly  to  compensate  for  the  problem  causing  conditions  with  avail¬ 
able  practices  (such  as  manual  or  automatic  testing,  additional  mainte¬ 
nance  man-hours  or  training,  additional  spare  or  replacement  parts,  etc.) 
than  it  was  to  restore  or  achieve  satisfactory  performance  levels  with  prac¬ 
tices  that  would  integrate  the  diagnostics  elements. 

The  interface  devices,  adapters,  software  programs  and  documentation  to 
test  individual  weapon  system  electronic  items  at  the  box  and  circuit  card 
level. 

“Items  that  can  be  used  directly  by  the  armed  forces  to  carry  out  combat 
missions  and  that  cost  more  than  $100,000  or  for  which  the  eventual  total 
procurement  cost  is  more  than  $10,000,000.  Such  term  does  not  include 
commercial  items  sold  in  substantial  quantities  to  the  general  public.  (See 
Title  10,  United  States  Code,  Section  2403,  ‘Major  weapon  systems:  con¬ 
tractor  guarantees.’)”  [DOD  91,  B-121] 
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AMID 

Aviation  Maintenance  Integrated  Diagnostics 

ASIS 

Advanced  Strike  Integrated  Diagnostics 

ATE 

Automatic  Test  Equipment 

ATS 

Automatic  Test  System 

BIT 

Built-In  Test 

CND 

Can  Not  Duplicate 

COTS 

Commercial  off  the  Shelf 

DOD 

Department  of  Defense 

IC&A 

Industrial  Capabilities  and  Assessments 

ID 

Integrated  Diagnostics 

IDA 

Institute  for  Defense  Analyses 

IEEE 

Institute  of  Electrical  and  Electronics  Engineer,  Inc. 

IMDS 

Integrated  Maintenance  Data  System 

IPT 

Integrated  Product  Team 

IRB 

Independent  Review  Board 

JAST 

Joint  Advanced  Strike  Technology 

MOE 

Measure  of  Effectiveness 

NSIA 

National  Security  Industrial  Association 

OJT 

On  the  Job  Training 

OSD 

Office  of  Secretary  of  Defense 

PMA 

Program  Management  Agreement 

POC 

Point  of  Contact 

PPBS 

Planning,  Programming,  and  Budgeting  System 

R&D 

Research  and  Development 

RTOK 

Re-Test  OK 

SPO 

System  Project  Office 

TPS 

Test  Program  Set 

UUT 

Unit  Under  Test 
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